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1.  Introduction 


Tlie  goal  ot  t  his  document  is  to  provide  practical  atlvice  on  how  to  improve  Vi>'\  on  A I 
prc^jects.  While  there  is  some  discussion  of  the  state  of  the  art  in  V'A:V  reseairh — and 
some  attempt  to  advance  the  state  of  the  art — the  emphasis  is  on  guidelines  and  tools 
that  can  provide  immediate  help  in  present  day  development  elforts.  We  assume  t  hat 
t lie  reader  is  familiar  with  standard  V&V'  practice,  as  reflected  in  textbooks  (<'.g..  [9. 
45] )  and  DoD  applicable  standards,  such  iis  2167A  and  2168.  The  fpiestion  we  ai  tempt 
to  answer  is:  How  can  I  apply  iny  knowledge  of  V'T-T  piactice  to  A  I  deo  lopnif-  iit. 
which  if  tins  eery  different  from  the  sorts  oj  examples  discussed  in  textbooks  and  which 
ciiniwl  he  rosily  mapped  into  the  lifecycle  models  of  thr  DoD  standards  '•! 

In  Part  1,  the  emphasis  is  on  laying  a  hrm  foundation.  Icrms  that  play  a  <  (Ui- 
tral  role  in  subsequent  discussion — including  'verification.'  'validation.'  and  '.Xrtifi- 
cial  Intelligence' — are  defined,  to  reduce  the  chances  of  misunderstanding  our  .idvice. 
•Also,  a  new  representation  of  system  lifecycles  is  presented,  which  we  believe  ymi  will 
find  useful  in  analyzing  your  organization's  .AI  development  efforts. 

In  Part  II.  the  focus  shifts  to  providing  advice.  The  part  lonsists  of  five  sections, 
one  arldressed  to  project  technical  leaders,  one  to  system  specifiers,  one  to  system 
designers,  one  to  programmers,  and  one  to  documenters.  Each  role  contributes  in  a 
unique  way  to  the  overall  V&:V  process,  and  so  we  present  a  set  of  guidelines  specific 
to  each  role. 

Part  Ill  consists  of  a  collection  of  three  appendices.  The  first  is  the  I'si'r's  Manual 
for  a  software  tool.  .ASP.  we  have  developed  to  support  the  X'lAV  proi  ess  by  allowing 
programmers  to  better  integrate  formal  testing  with  code  development.  Our  ho|)e  is 
that  the  tool  increases  the  value  of  formal  testing  so  much  that  [)rogrammers  will  see 
it  as  a  help  rather  than  a  burden.  The  second  appendix  is  a  glossary  of  \  A'\  terms. 
The  final  appendix  is  a  guide  to  commercially  available  C.A.Sb  tools. 
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Part  I 

V&V  AND  AI 


2.  What  V&:V  Is 


is  short  tor  ’’verilication  and  validation. "  The  ol>jectivr  of  tliis  chapter  is  to 
gtiiiraiitee  that  we  have  a  coimnon  iiiulerstaiiding  ofjiist  what  tlie  words  ’vfi  ilicatioii" 
and  "validation.’  mean  as  they  are  used  in  the  report.  "VeriHcatiori”  in  particular  is 
iiserl  ill  a  number  of  different  ways  by  software  developers.  (Some  rh'liates  about  tin' 
utility  of  verification  in  .\1  soitware  development  could  be  settled  simply  l)\  having 
the  participants  explain  wliat  they  meant  by  "‘verificatioir.) 

Verification  and  validation  are  complementary  software  development  at  tiviiu's.  <‘acli 
coiUributes  to  software  (juality.  ['erification  is  the  process  of  assuring  inti'inal  con¬ 
sistency  in  software  development,  while  validation  is  the  |)rocess  of  assuring  that  tin' 
soitware  being  develope<i  satisfies  its  reriuirements.  .A  i)opular  way  o|  'avmg  this  is 

\  e nfirntion.  nssnr&s  that  the  -'oftirnre  i-a  built  nijht:  rnlidntion  that 

the  mjht  ■■^oftiriire  /.s  hiult. 

One  important  form  of  verification  is  assuring  that  the  various  products  generated 
during  software  development  process  are  consistent.  Let  us  call  this  mh  r- product 
I'f  nficntion.  In  virtually  every  effort  to  develop  software  for  use  by  sonn'one  other 
than  the  author,  there  are  multiple  products  of  the  development.  .\t  a  minimum, 
there  is  the  software  and  some  rlescription  of  how  to  use  it.  In  this  case,  inter-product 
verification  amounts  to  making  sure  that  the  description  correctly  land  completely! 
describes  the  interface  to  the  delivered  software.  But  typically,  especiallv  in  large 
development  efforts,  there  are  many  more  products  of  software  development.  In 
addition  to  the  software  itself,  these  may  include 

•  a  statement  of  the  requirements  that  the  software  must  satisfy. 

•  a  specification  of  properties  of  the  software  system — its  functionality,  real-time 
behavior,  and  so  forth. 

•  a  design  document,  or  even  multiple  design  documents  at  varying  levels  of  rletaii 
and  employing  a  variety  of  design  formalisms. 

•  source  code  for  the  software. 

•  a  description  of  test  procedures  and  test  cases,  at  both  the  system  and  subsystem 
level. 

•  additional  software  that  supports  testing. 

•  a  user  s  manual,  and 


•  a  maintainer's  manual 


Thus,  inter- product  verification  often  retjuires  making  sure  that  a  system  with  the 
specified  properties  will  satisfy  the  state<l  requirements,  that  the  design  (orrectly 
elaborates  the  specification,  that  the  source  code  correctly  implements  the  design, 
and  so  on. 

.4  second  form  of  verification  is  ini m-protluct  rtrification.  assuring  that  each  product 
of  software  development  is  int<'rnally  consistent.  A  canonical  e.xample  from  .\1  is  val¬ 
idation  of  a  knowleilge  ba.se.  If  a  knowledge  ba.se  consists  of  logical  formulas  that  are 
intended  to  l)e  true  of  the  application  domain,  that  knowledge  base  should  be  logi¬ 
cally  consistent,  for  a  logically  inconsistent  set  of  formulas  must  contain  at  h'ast  one 
formula  that  is  false.  Moreover,  since  any  formula  whatsoever  can  be  derived  from  an 
inconsistent  set  of  formulas  by  correct  rea.soning,  the  usual  justification  for  accf'pt- 
ing  the  conclusions  derived  from  the  knowledge  bcise — that  the  inference  merlianism 
reasons  correctly — has  l)een  undermined.  A  system  with  an  inconsistent  knowleflge 
l)ase  is  generally  useless,  so  knowledge  base  verification  includes  as.siiring  consistencv. 
It  may  also  include  assuring  that  the  knowledge  base  has  other  properties — say.  that 
none  of  the  formulas  can  lie  derived  from  others — judged  to  be  desirable.  The  t  onsis- 
tency  criterion  can  be  applied  to  most  lifecycle  products:  a  requirements  statement 
should  be  consistent  in  the  sense  that  there  is  no  property  that  it  both  requires  and 
forbids  the  system  to  have;  a  s\stem  specification  should  be  consistent  in  the  sense 
that  there  is  no  property  that  it  guarantees  the  system  will  both  have  and  lack:  and 
so  on.  .Just  as  in  the  case  of  the  knowledge  base,  there  are  usually  other  acceptability 
criteria  for  the  products  that  must  be  assured  as  well.  .4  typical  example  is  assur¬ 
ing  that  source  code  meets  applicable  coding  standards.  The  dividing  line  between 
intra-product  verification  anc'  inter-product  verification  is  somewhat  arbitrary — if  rhe 
design  is  documented  as  a  series  of  successive  refinements,  checking  that  a  purported 
refinement  is  in  fact  a  refinement  is  intra-pro<iuct  verification  if  the  entire  series  is 
thought  of  as  a  single  product.  l)ut  is  inter-product  verification  if  each  design  in  the 
series  is  thought  of  as  a  product — Imt  useful  in  practice,  especially  on  larger  projects 
where  different  products  are  produced  by  different  groups  of  developers. 

.4  common  error  in  discussions  of  V4:V  is  to  confuse  verification  against  a  statement 
of  requirements  and  validation  against  tiie  true  requirements.  Generally,  software  is 
written  to  solve  some  problem,  and  the  people  who  create  the  software  that  is  intended 
to  solve  the  problem  are  not  the  people  who  have  the  problem.  To  be  successful,  the 
software  developers  must  solve  the  problem,  but  they  must  also  .satisfy  the  terms  of 
the  contract,  formal  or  informal,  with  their  <  lients.  If  the  clients  have  a  good  under¬ 
standing  of  what  they  want  in  a  solution,  the  contract  might  contain  a  statement  of 
requirements.  This  is  the  sort  of  ca.se  where  it  is  particularly  tempting  to  say  that 
validation  consists  of  making  sure  the  software  satisfies  the  stated  requirements.  But 
it  is  still  important  to  distinguish  between  the  stated  requirements  and  the  actual  re¬ 
quirements.  because  satisfying  the  stated  reciuirements  may  not  be  enough  guarantee 
that  the  problem  will  be  solved.  Ideally,  validation  consists  of  using  the  system  to 
actually  solve  the  problem,  or  at  least  representative  instances  of  a  general  problem. 
If  this  is  impossible,  weaker  forms  <d  validation,  such  as  running  the  system  in  some 


1 


sort  oi'  testbed  enviroiuuent  on  simnlftted  data,  inay  have  to  suffice.  When  llie  re- 
(juirements  statement  is  an  input  to  the  development  process  rather  than  a  product  .d 
the  de\'eiopment  process,  clu'cking  that  the  state<l  re(|uirements  are  satisfic’d  cannot 
l)e  considered  to  he  verification.  It  can  be  considered  validation,  but- -here  is  the 
main  point! — it  is  a  very  weak  Form  oF  validation.  .A.  system  cannot  be  irfi!  validated 
simplv  on  the  basis  oF  stnfol  requirements,  because  discovering  and  correctly  statinu 
(at  a  low  enough  level  oF  detail  that  satisfaction  can  be  effectively  cleterminerl)  all  the 
reciuireiuents  For  solving  a  problem  is  tremendously  difficult.  For  the  problems  typi¬ 
cally  addressed  by  .\I.  produc  ing  a  •‘completely  adequate"  statement  oF  refiuiremeut-> 
is  impossible.  From  a  practical  point-of-view. 

Validation  oF  a  system  can  oidy  be  achieved  by  '‘trying  it  out.”  Other  devplopment 
products  can  be  validated  to  a  limited  c’xtent.  by  trying  them  out— For  example,  .m 
executable  Functional  specification  can  l)e  validated  by  using  it  as  a  prototype.  ,md 
a  users  manual  can  lie  validated  by  having  a  prospecti\e  user  attempt  to  run  'lie 
system  bv  referring  to  the  manual — or  indirectly  in  conjunction  with  verification.  In 
Fact,  a  common  method  oF  validating  a  requirements  statement  is  to  develop  soltwan* 
that  has  been  verified  to  (at  least  partially)  implement  it.  and  then  validating  the 
prototype  through  use.  L'hus.  an  attempt  to  validate  the  software  system  by  reference 
to  iho.se  stated  requirements  has  the  matter  reversed.  Once  the  system  is  •'ufficientl.v- 
complete  that  direct  validation  though  use  is  possible,  the  requirements  statement 
should  play  no  Further  role  in  system  validation,  because  any  conflict  between  the 
'tated  recjairements  and  the  system  should  be  arbitrated  by  direct  valirlafion  against 
the  true  requirements,  since,  in  most  cases,  the  stated  requirements  are  as  likely  r.j 
lie  wrons  as  the  software.  The  bottom  line  is;  If  the  system  solves  the  problem, 
it  doesn’t  matter  that  the  requirements  statement  was  wrong.  (Of  course  this  doe- 
not  mean  that  the  reiiuirements  statement  need  not  be  corrected.  The  requiremeni  s 
-tatement  plays  an  important  role  during  system  maintenance,  since  it  provides  guid- 
aiK  e  on  what  can  and  cannot  be  changed.  It  should  always  repre.sent  the  best  curv^ni 
understanding  of  the  true  requirements.)  So,  while  the  system  may  derive  signifi¬ 
cant  indirect  validation  during  development  by  virtue  of  its  verified  satisfaction  ot 
rhc’  partially  \alirlated  requirements  statement,  the  Fundamental  system  validation 
activitv — (iirect  validation  of  the  system  through  use — provides  further  iiulirect  vali¬ 
dation  of  the  stated  requirements.  That  is.  once  the  system  is  running,  flie  system  :> 
n-ed  to  validate  the  requirements  statement  rather  than  the  reverse. 


) 


3.  What  AI  Is 


■'AI'’  is  short  for  "Artificial  latolligence."  So  far,  so  good.  Bui,  judging  from  the 
I  it. stature,  few  people  have  ever  agreed  on  a  definition  of  "Artificial  Intelligence."  For 
our  purposes,  it  is  most  convenient  to  adopt  a  version  of  the  position  that  .A.I  is  what 
.\I  people  do.  Since  our  concern  is  with  software  systems,  this  amounts  to  identifying 
.\I  with  a  type  of  program,  namely,  those  which  use  the  programming  iec/ind/ae.s 
used  in  paradigmatic  .\I  programs,  embedded  in  an  arclnitctiirt  chc:^' acteristic  of 
paradigmatic  .AI  programs. 

This  position  proves  convenient  because  the  only  features  of  a  program  that  much 
influence  the  choice  of  methods  is  the  collection  of  programming  rechnicpies 

(Uiiployed  and  the  architecture.  .Many  other  factors  are  relevant  to  t  he  choice — ^ranging 
from  the  type  of  problem  addressed,  through  the  lifecycle  model  used  to  guide  the 
development,  to  contractual  obligations — but  these  are  external  to  the  program.  It 
turns  out  that  these  other  factors  are.  effectively,  less  relevant  to  the  choice  if  we 
restrict  our  attention  to  .AI  software,  since  most  .AI  systems  address  problems  that 
are  similar  in  the  relevant  respects,  most  .AI  development  efforts  use  lifecycle  models 
that  are  similar  in  relevant  respects,  and  so  on.  So  in  this  Chapter,  we  will  enumerate 
some  of  the  programming  techniques  that  make  a  system  .A I.  categorize  some  .A I 
architectures,  and  then  look  at  the  common  factors  in  .AI  system  development  that 
influence  the  choice  of  V&;V  methods.  Finally,  we  comment  on  the  potential  for  using 
W  techniques  to  support  V\CV  of  AI  systems. 

3.1  AI  Programming  Techniques 

The  following  list  was  largely  extracted  from  a  standard  text  on  .A  I  programming  tech¬ 
niques.  Charniak.  et  al.'s  Artificial  Intelligence  Programming  [13].  a  good  overview 
of  the  subject.  (.Another  good  source  on  the  subject  is  .N’orvig's  Paradigms  of  A I 
Programming  [33].) 


3.1.1  Data-driven  Programming 

.\ttaching  programs  to  data  and  deciding  what  to  do  by  retrieving  aiul  running  pro¬ 
grams  associated  with  data  is  data-driven  programming.  One  common  example  is  use 
of  message  passing  to  invoke  methods  associated  with  the  message.  .As  this  example 
illustrates,  data-driven  programming  is  not  restricted  to  .A I.  However,  it  tends  to  l^e 
especially  common  in  .AI,  because  .AI  programs  are  often  written  in  languages,  such 
as  Lisp  and  Prolog,  that  allow  programs  to  be  treated  as  data.  This  capability 
facilitates  storing  code  in  comparatively  complex  data  structures,  such  as  a-lists  and 
hash  tables,  and  so  encourages  use  of  data-driven  techniques.  (.Another  reason  for 
including  this  technique  in  our  list  is  that  use  of  data-driven  programming  stronglv 
influences  the  choice  of  Vi^:V'  methods.) 


fi 


3.1.2  Discrimination  Nets 


A  common  programming  problem  in  Ai  is  the  classification  of  information  based  on 
tests  of  its  properties.  An  ii  itial  test  is  applied;  based  on  the  result,  another  test 
may  l)e  chosen  and  applied:  ba.sed  on  the  result,  another  test  may  be  chosen  and 
applied:  and  so  on.  Thus,  the  collection  of  tests  can  be  thought  of  a.s  defining  a 
network,  with  a  link  from  one  test  to  another  whenever  the  result  of  applying  tlie 
former  can  trigger  the  application  of  the  latter.  Such  a  network  is  a  Hiscrinuiiatioii 
net.  good  e.xample  of  applying  this  technic[ue  to  an  AI  problem  can  be  found  in 
•Appendi.x  .A.  which  describes  an  Al-based  V&V  tool  developed  at  .ADS  that  einpio\  > 
a  special-purpose  language  for  defining  discrimination  nets  for  software  testing. 

3.1.3  Meta-level  Control  Structures 

.A  comnion  form  of  data-dri\  pn  programming  in  .AI  is  the  use  of  a  clo.sure  to  represent 
the  state  of  a  suspended  process.  .A  collection  of  loosely  related  techniques  —  iKjenda- 
based  control,  queue-based  control,  streams,  coroutines,  possibilities  lists,  and  so  on — 
have  been  based  on  this  idea.  The  approach  is  generally  useful  when  ineta-level 
reasoning  about  control  is  performed.  Object-level  operation  (which  may  be  search 
of  a  game  tree,  adding  information  to  a  blackboard,  refining  a  plan,  or  just  about  any 
other  typical  object-level  activity)  is  suspended,  a  ‘‘next  move'*  is  determined,  and 
the  suspended  operation  is  then  resumed.  LiSP  includes  a  variety  of  constructs  that 
support  this  technique,  including  COMMON  LiSP’s  closures  and  Scheme's  first-ciass 
continuations,  and  expert  system  shells  and  other  higher-level  program  development 
tools  frequently  support  some  form  of  meta-level  reasoning  via  this  rechnicpie. 


3.1.4  Deductive  Information  Retrieval 

.Many  .AI  applications  derive  information  from  facts  stored  in  some  sort  of  knowledge 
base.  .Such  applications  can  be  thought  of  as  "smart "  databases,  capable  of  retrier  niii 
not  only  information  that  has  been  stored  in  them  explicitly,  but  also  of  retrievin»: 
information  implicit  in  the  stored  facts.  Such  systems  are  said  to  perform  dr  duct  no 
information  retrieval.  ‘‘Deduction'  is  used  rather  broadly  here,  to  include  not  only 
strict  logical  deduction,  or  even  non-moiiotonic  inference  procedures  modeled  on  log¬ 
ical  deduction,  but  also  the  ad  hoc  heuristic  procedures  used  in  semantic  networks. 
Deduction,  in  this  generic  sense,  is  any  inference  procedure  applied  to  the  explicitly 
represented  information. 

It  may  seem  that  the  V&:V  methods  appropriate  to  logical  deduction  and  those  appro¬ 
priate  to  looser  forms  of  inference  would  be  quite  different,  making  further  subdivision 
of  this  technique  useful.  However,  in  practice,  even  strictly  deductive  inference  pro*  e- 
dures  are  incomplete — and  sometimes  even  unsound — theorem  provers,  for  the  sake 
of  efficiency.  To  pick  the  most  wirlely  known  example,  stantlard  PROLOG  employs  an 


unsound  algorithm  tor  unification;  the  so-called  “occurs  check”  is  omitted  in  order  lo 
make  unification  of  a  term  with  a  variable  0(  1)  rather  than  0(n)  in  the  length  ot  the 
term,  just  as  assignment  is.  Without  this  deviation  from  "logical  purity."  Prolog 
could  not  compete  in  efficiency  with  conventional  languages.  .A  clever  programmer 
can  easily  arrange  things  so  that  no  incorn'ct  conclusions  are  derived,  and.  clearly, 
this  is  exactly  the  sort  of  thing  that  V^V  procedures  should  check.  Therefore,  rather 
than  attempting  to  subdivide  the  clas.s  of  deductive  information  retrieval  techni(|Mes. 
we  will  focus  on  assigning  \'LV  techniques  to  these  systems  based  on  properties  that 
the  inference  method  is  intended  to  possess. 

3.1.5  Production  Systems 

.A  production  system  consists  of  a  collection  of  condition-action  rules,  the  production 
memory,  and  a  data  store,  the  working  memory.  Its  operation  is  coiitrolletl  Ijv  a 
recognize-nct  loop:  a  rule  whose  condition  is  true  is  found,  and  the  corresponding 
action  is  performed.  The  action  generally  includes  changes  to  the  working  memory 
that  influence  which  conditions  are  satisfied  on  the  next  cycle.  Thus  production 
systems  use  a  natural  generalization  of  forward-chaining  inference.  .A  number  of 
widely-used  expert  system  tools,  such  as  OPSo.  support  building  production  systems. 

3.1.6  Frame  Databases 

.A  frame  is  a  data  structure  u.sed  for  knowledge  representation  that  naturally  gener¬ 
alizes  record  structures  and  LiSP's  property  lists.  .According  to  Minsky  [31]. 

[w]e  can  think  of  a  frame  as  a  network  of  nodes  and  relations.  The  "top 
levels”  of  a  frame  are  fixed,  and  represent  things  that  are  always  true 
about  the  supposed  situation.  The  lower  levels  have  many  terminals — 
"slots”  that  must  be  filled  with  specific  instances  or  data. 

Frames  have  been  specialized  for  particular  purposes;  Shank  and  .Abelson's  scripts  [37]. 
used  for  natural  language  understanding,  are  a  good  example  of  specialized  frames. 

Just  as  in  the  case  of  production  systems,  frame  databases  provide  a  general  purpose 
knowledge  representation  that  is  the  basis  of  several  popular  expert  system  building 
tools,  such  as  KRL  and  FRL.  (Frames  and  production  rules  combine  in  a  natural 
fashion,  and  many  expert  system  shells  provide  both  techniques.) 

3.1.7  Backtracking 

Search,  in  one  form  or  another,  is  central  to  .Al.  The  most  common  search  paradigm 
is  to  procoeel  depth-first,  that  i.s.  to  atfetupt.  to  find  a  path  through  the  search  nee 
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starting  from  the  top  and  exploring  a  single  branch  at  a  time.  For  example,  a  planning 
system  might  attempt  to  refine  a  plan  by  .selecting  among  various  refinement  opera¬ 
tors,  choosing  the  operator  according  to  some  principle  ranging  from  the  simple  and 
inexpensive  (such  as  choosing  the  first  applicable  operator  from  the  list  of  operators) 
to  the  complex  and  costly  (such  a.s  calculating  some  measure  of  which  operator  is 
"best"  among  all  applicable  operators  based  on  detailed  consideration  of  the  current 
state  of  the  plan).  Domain-specific  knowledge — sometimes  represented  explicitly,  and 
sometimes  implicitly  as  an  operator  ordering  or  a  numerical  formula  ii.sed  in  comput¬ 
ing  an  operator  quality  metric — is  used  to  guide  the  search.  If  choosing  the  wrong 
Itranch  ot  the  search  tree  (e.g.,  choosing  the  wrong  refinement  operator)  can  lead  to 
a  d<’ad-end.  some  form  of  backtracking  is  employed. 

In  the  simplest  case,  chronological  backtracking,  the  most  recent  decision  is  (  hanged: 
state  changes  associated  with  actions  performed  a,s  a  result  of  that  decision  are  un- 
doite,  and  a  different  decision  is  made,  resulting  in  the  exploration  of  an  alternative 
branch  of  the  search  tree.  But  if  the  system  tracks  the  reasons  for  each  decision  and 
can  analyze  the  cause  of  failure,  a  more  intelligent  form  of  backtracking,  de pt  n dtncy- 
dincted  backtracking,  can  b<>  employed.  Rather  than  ba.cktracking  to  the  most  recent 
decision,  which  may  have  been  irrelevant  to  the  failure,  control  pa.sses  back  to  a  de¬ 
cision  point  that  is  certainly  relevant.  .Moreover,  the  system  may  use  sophisticated 
reason  maintenance  facilities  to  avoid  having  to  undo  all  state  changes  as.sociated 
with  the  failed  branch;  some  of  the  decisions  made  after  the  decision  being  reconsid¬ 
ered  may  be  essentially  independent  of  that  decision,  and  so  need  not  be  reconsidered. 
This  might  be  thoughts  of  as  a  jump  across  the  tree  rather  than  "backtracking;"  see 
Fig.  ;M. 

The  impact  of  particular  programming  techniques  on  choice  of  \'ScV  methods  is  dis¬ 
cussed  in  Section  S. 

3.2  AI  Programming  Tools 

.A I  programming  tools  span  the  range  from  programming  languages  especially  suited 
to  .\I  programming,  through  high-lev'el  programming  environments,  to  knowledge- 
based  system  shells.  Choice  of  tools  hcis  sonie  effect  on  appropriateness  of  \\A:V 
methods,  so  we  will  briefly  summarize  some  relevant  features  here  to  prepare  for  the 
discussion  of  the  impact  of  tool  choice  on  V in  Sectioti  o. 

3.2.1  AI  languages 

Lisp  is  really  a  family  of  programming  languages,  based  on  a  common  core  of  ideas, 
that  has  been  the  primary  vehicle  of  .M  research  for  over  30  years.  LiSP  was  designed 
tor  symbolic,  rather  than  numerical,  programming,  and  has  grown  over  the  years 
to  include  precisely  those  features  most  useful  for  .\I  programming.  For  example. 
Lisp  has  a  flexible  type  mechanism  that  makes  it  easy  to  alter  and  expand  data 


Figure  M:  Varieties  of  Backtracking 


repiT'sentalions.  a  feature  of  obvious  utility  in  incremental  development  of  proiotyi)e 
systems.  The  fact  that  programs  are  represented  as  data  structures  makes  it  ca-sy  lo 
develop  "meta-level"  facilities,  including  program  development  tools.  Every  read«*r 
of  this  manual  is  probably  familiar  with  the  basics  of  LiSP.  The  l)est  source  for  a 
detailed  e.vplaiiation  of  why  Lt.SP  is  special,  and  especially  suited  to  .M.  is  Allen's 
Anatomy  ol  IJSP  [2|. 

Compared  to  LiSP.  Proi.OC:  is  a  newcomer  on  the  .\I  scene,  being  only  about  JO 
years  old.  Pltf)LOG  \va.s  a  lirst  attempt  at  realizing  the  itleal  of  logic  programming, 
i.e..  treating  computation  as  deduction.  PROLOG  can  be  thought  of  a.s  an  ellu  ieut 
constructive  theorem  prover  for  a  subset  of  first-order  logic.  .A.s  a  result.  PR()I.<;<:.  has 
a  firm  logical  foundation  to  support  the  analysis  of  programs,  in  rather  sharp  i  (jui  rasi 
to  Lisp.  But.  more  simple.  PhoI.OG  can  be  thought  of  as  the  result  of  supply im*  the 
piogrnni  chni.st 

Pi.r.ii)  if(J||.r.ri)  and  C^n(.r.r.i). 

with  a  prorfdainl  irtiding: 

For  any  . the  goal  of  proving  P(x.i;)  can  be  satisfied  bv 

proving  . and  Qn(J.-n). 

and  using  a  depth-first  search  strategy  based  on  the  ordering  of  program  claus<’s  aii<l 
chronological  backtracking  to  find  a  reduction  of  a  given  goal  to  formulas  storc-d  in  a 
knowledge  base.  Sterling  and  Shapiro's  The  Art  of  Prolog  [42}  provides  a  nice  nii.\-  .d' 
the  theory  and  practice  of  PROLOG. 

The  claim  that  PROLOG  was  a  reasonably  efficient  general  purpose  programming 
formalism  was  met  with  consirlerable  scepticism  in  the  United  States,  based  on  ex¬ 
perience  with  some  special  purpose  .AI  languages  in  the  early  1970'3.  MlCROPL.AX- 
XER  in  particular,  ba.sed  on  similar  ideas.  .As  a  result.  PROLOG  was  largely  ignoreri 
here  until  the  .Japanese  announced  that  it  would  play  a  central  role  in  their  Fifth 
Generation  effort.  (However.  LiSP  was  largely  ignored  in  Europe  fluring  this  pe¬ 
riod.  because  it  was  Judged  too  costly  in  machine  resources  consumerl.  and  Proi.o<; 
was  widely  experimented  with.)  .After  the  announcement,  it  was  discoveri-d  that 
some  significant  breakthroughs — Warren's  approach  to  compilation  being  pariit  ular 
noteworthy — combined  with  Prolog’s  willingness  to  sacrifice  logical  purity  for  elli- 
ciency.  had  resulted  in  a  programming  language  very  well  suited  to  some  .Al  problems. 
In  fact,  it  turns  out  that  many  of  the  programming  techniques  listed  above  can  lie 
implemented  in  PROLOG  in  a  perfectly  straightforward  fcishion  [7]. 

Both  Lisp  and  Prolog  have  meta-level  introspection  facilities  that  support  a  deeper 
level  ot  computntioniil  introspection  and  reflection  [39]  than  conventional  languages, 
such  as  ('  and  .Ada.  This  fact  has  some  impact  on  the  scope  and  limits  of  <  t'rtam 
V'.itV  approaches:  see  Section  8. 


3.2.2  Higher  level  tools 


Higher  l('vel  tools  can  l)e  (roughly)  divided  into  high-level  programming  environments 
and  expert  system  shells.  .A  shell  is  typically  the  result  of  abstraction  from  an  existing 
know  ledge- based  system.  It  consists  of  an  inference  engine,  an  empty  knowledge  base 
that  is  |)opulated  for  the  particular  application,  and  some  support  tools.  .\  hii/h- 
lecel  enrtronmenl  is  a  collection  of  tools — inference  engines,  knowledge  representation 
languages,  and  so  on — that  have  been  at  least  partially  integrated,  so  that  the  user 
can  choose  among  the  various  options.  Emycin  is  the  classic  example  of  a  shell.  I)ut 
more  are  constantly  becoming  available,  often  specialized  to  a  particular  domain  so 
as  to  supply  even  greater  leverage.  A  good  example  is  Verity's  TOPIC  system  for  text 
letrieval:  the  developer  provides  a  definition  of  the  concepts  to  be  used  in  ret  i  ieval  and 
the  text  database,  and  TOPIC  provides  everything  else.  S.l,  KeE.  .ART.  ProK.\P(’.\. 
and  so  on.  are  all  representative  examples  of  high-level  environments. 

But  for  both  shells  and  high-level  environments,  the  impact  of  the  tools  mi  (  hoice 
of  V&V  methods  depends  only  on  those  programming  techniques  made  available'  bv 
the  tool  that  are  used  by  the  developer.  No  direct  matching  of  methods  lo  tools  is 
necessary  (or.  in  the  case  of  environments,  desirable).  This  is  fortun~ne.  as  the  most 
popular  tools  at  the  time  you  are  reading  this  manual  are  likely  to  be  dilFerent  from 
the  ones  available  when  it  was  written. 

3.3  A  Categorization  of  Knowledge-Based  System  Architectures 

(i’ategorization  of  .A I  architectures  in  general  is  a  difficult  task,  which  will  [\ot  be  ,ii- 
temptecl  in  this  manual.  Even  paradigmatic  .AI  systems  can  empliA’  ad  ho(  architec¬ 
tures  that  seemingly  cannot  be  described  in  any  way  that  provides  guidanci*  in  detimng 
a  verification  and  validation  methodology.  However,  categorization  of  a  K'stricted  - 
but  important! — subclass  of  .AI  architectures,  those  employed  in  knowledge-basetl 
systems,  is  feasible.  For  our  purposes,  a  know  ledge- based  system  is  one  that  can  bp 
divided  into  a  knowledge  base,  an  inference  engine  that  derives  conclusions  <  perhaps 
together  with  explanations)  from  the  knowledge  base,  and  support  software  !<uch  as 
the  user  interface).  The  knowledge  base  might  consist  of  logical  assertions.  i  ondition- 
action  rules  (perhaps  vvith  associated  confidences),  a  Bayesian  network,  or  any  other 
natural  representation  of  domain  knowledge.  The  inference  engine  can  be  .irbiirar- 
ily  complicated,  from  a  simple  algorithmic  scheme  such  as  backward  chaining  from 
a  goal  or  propagation  of  confidences  in  a  Bayesian  or  qutisi- Bayesian  fashion,  to  a 
know  ledge- based  system  in  which  a  ’“meta-lever  inference  engine  determines  how  to 
control  use  of  the  ’’object- level”  domain  knowledge  based  on  the  contents  of  a  sepa¬ 
rate  ’’meta-level"  knowledge  base  that  contains  an  explicit  representation  of  control 
knowledge.  .Much  of  the  second-generation  knowledge- based  expert  system  work  in 
the  early-to-mid  l9S0’s  was  driven  by  the  recognition  that  it  was  ustdul  to  factor  out 
an  explicit  representation  of  the  control  knowledge,  because  doing  so 


•  increases  modularity,  which  makes  the  system  ocisier  to  develop  and  niodil'y  [IB. 

14.  1], 

•  makes  the  domain  knowledge  more  reu-sahle  [14], 

•  facilitates  explanation  generation  [46],  and 

•  places  the  focus  on  "representational  adetiuacy"  rather  than  on  elficiency  in 
representing  domain  knowledge  [19|. 


Verification  and  validation  of  simple  inference  engines  is  straightforward.  It  sliouki 
be  proved  that  the  inference  algorithm  produces  the  desired  conclusions.  That  the 
code  correctly  implements  the  algorithm  should  be  verified  using  conventional  verifi¬ 
cation  techniques.  The  system  should  be  validated  by  exercising  it  on  represenlativ(* 
and  extremal  knowledge  sets.  That  is.  a  simple  inference  engine  is  verified  and  vali¬ 
dated  as  if  it  were  "just  another  algorithm.”  There  is  no  special  ".\I  V\V:V  '  problem 
involved.  Therefore,  the  taxonomy  of  know  ledge- based  systems  is  heavily  slanted 
towards  more  complex  inference  procetlures,  where  special  problems  arise.  The  cat¬ 
egorization,  adapted  from  the  one  in  [19],  is  based  on  how  much  effort  is  put  into 
determining  the  next  inference  step  or  series  of  inference  steps,  and  on  when  this 
"meta-leveT’  inference  is  performed.  This  categorization  ignores  factors — such  as. 
whether  the  inference  is  performed  at  a.ssert ion- time  or  at  inference-time — that  are 
essential  to  design  of  the  system,  but  are  largely  irrelevant  to  how  \'L\'  should  be 
performed. 

Meta-level  inference  architectures  are  distinguished  by  the  fact  that,  at  any  given 
time,  the  system  can  be  active  at  either  of  two  levels.  It  can  be  determining  what  to 
do.  by  drawing  conclusions  from  the  meta-level  control  knowledge,  or  it  can  be  dointj 
it.  by  drawing  conclusions  from  the  object-level  domain  knowledge.  Thus,  there  is 
a  spectrum  of  "how  meta”  a  meta-level  inference  architecture  is.  depending  on  how 
much  time  it  spends  working  at  the  meta-level,  from  systems  that  perform  very  little 
rneid-level  inference  to  systems  that  perform  meta-level  inference  almost  exclusively. 
.A  system  that  spends  very  little  time  working  at  the  meta-level  will  be  called  a  object- 
level-onented  inference  architecture.  .\  system  that  expends  the  bulk  of  its  effort  at 
the  meta-level  will  be  called  a  rnetn-lecfl-onmttd  inference  nrchitectuve .  system 
that  puts  substantial  time  and  effort  in  at  both  levels  will  be  called  a  nnred-lerel 
inference  architecture. 

.Mi.xed-level  architectures  will  be  further  subdivided  according  to  the  conditions  under 
which  resources  are  devoted  at  the  meta-level.  .Most  common  are  reflect-nnd-act  sys¬ 
tems.  which  perform  meta-level  reasoning  before  (or.  equivalently,  after)  each  object- 
level  step.  Blackboard  systems  commonly  use  a  reflect-and-art  loop  to  apply  control 
knowledge  in  a  particular  situation.  If  meta-level  reasoning  is  used  only  when  some 
crisis  develops  at  the  object-level — where  a  crisis  can  be  anything  from  having  too 
many  options  available  to  recognizing  an  inconsistency  in  the  object-level  knowledge 
ba.se — we  have  a  crisis  rnnnngemcnt  system.  Finally,  if  the  primary  function  at  the 


meta-level  is  to  divide  inference  tasks  into  subtasks,  which  are  then  handled  at  tlie 
object-level,  we  have  a  subtnsk  mnnntje/nenJ  system.  This  completes  our  taxonomy, 
which  is  graphically  represented  in  Fig.  3-2:  the  influence  of  particular  architecture 
categories  on  choice  of  V.SjV  techniques  is  discussed  in  Section  7. 

3.4  Common  Characteristics  of  Typical  AI  Problems 

As  was  mentioned  above,  the  nature  of  typical  problems  addressed  by  A I  ha.s  some 
impact  on  how  to  perform  V'&V  of  .\I  systems.  There  are  two  principal  relevant 
characteristics. 

1.  I'he  system  re(|uirements  are  hard  to  define.  This  is  partly  becau.se  of  the  ^i/e 
of  the  problems  addressed;  Al  is  often  applied  to  problems  too  large  to  deal 
with  by  more  conventional  algorithmic  techniques  guaranteed  to  produce  an 
optimal  solution.  It  is  partly  due  to  the  intrinsic  nature  of  the  problems;  .\I 
is  often  applied  when  the  problem  is  too  vague  to  be  addressed  by  tecitniques 
that  operate  on  precise  measurements.  In  either  case,  it  may  be  impossible  to 
define  what  counts  as  a  solution  to  the  problem  being  addressed,  frm  nfff  r  the 
system  has  been  completed.  There  just  are  no  cut-and-dried  criteria  that  can  be 
applied  to  determine  whether  the  system  "does  the  job.” 

2.  The  solution  process  is  knowledge  intensive.  Sometimes  computers  are  used  to 
solve  problems  involving  tedious  and  repetitive,  but  ultimately  simple,  calcula¬ 
tions  of  some  sort.  .\'o  real  understanding  of  the  problem  domain  is  needed:  a 
human  could  solve  the  problem  without  knowing  where  the  numbers  came  from 
or  what  they  represent.  Solving  an  .Al  problem  more  often  requires  a  detailed 
formal  model  of  the  domain  that  captures  complex  interdependencies  among 
various  domain  elements.  This  knowledge  can  be  hard  to  break  no  into  easily 
digestible  pieces,  which  makes  many  of  the  programmer's  tools  for  managing 
complexity  inapplicable.  .A  complicated  domain  model  can  be  harder  to  debug 
than  the  same  number  of  lines  of  "spaghetti  code.” 

Both  these  characteristics  are  illustrated  by  considering  some  typical  .\I  problem 
domains: 

•  planning  problems  too  large  to  handle  with  the  methods  of  Operations  Re¬ 
search,  involving  many  soft,  evolving  constraints  and  imperfect  knowledge  of 
the  conditions  under  which  the  plan  will  be  executed. 

•  image  and  signal  understanding  problems  that  require  discovery  and  analysis 
of  subtle  patterns  in  reams  of  data. 

•  natural  language  understanding  (no  more  need  be  said  about  this  one!). 


•  medical  diagnosis  (ditto!). 


and  so  on. 

The  impact  of  tlie.se  characteristics  on  \  will  be  considered  in  S<'<  tion  1. 

3.5  Common  Characteristics  of  the  AI  Software  Lifecycle 

The  most  important  characteristic  of  the  AI  lifecycle  is  that  systems  typically  evolve 
through  refinement  ol  a  [)rototype  that  is  used  to  help  determine  system  rec|uirements. 
Even  in  cases  where  sotne  of  the  code  can  be  developed  in  accordance  with  some  sort 
of  waterfall  model,  other  parts  are  constantly  evolving.  Thus,  even  more  than  in 
conventional  systems,  there  must  be  an  emphasis  on  reverification  and  revalidalion 
of  modilied  systems.  The  impact  of  tliis  emphasis  on  choice  of  VA'V  methods  will  be 
discussed  in  greater  detail  in  .Section  1. 


16 


4.  Development  Models  and  Methods 


Is  defining  a  single,  uniform  standard  A I  development  methodology  desirahle?  A1 
development  efforts  vary  too  iiinch  to  impose  strong  methodological  restrictions,  and 
weak  methodological  restrictions  are  useless.  However,  that  does  not  mean  that  no 
guidance  can  be  provided.  That  guidance  will  be  in  the  form  of  advice  on  how  to 
determine  an  appropriate  methodology  fora  project,  rather  than  methodological  rules 
that  should  always  be  followed.  Here  is  an  example  of  such  a  ■■metamethodologi(  al 
rule.”  one  that  we  urge  you  to  adopt. 

One  of  the  fir.'it  Insks  on  every  project  should  be  to  define  the  syslertt  <l(- 
relopinent  iiit thodoloyy  that  trill  be.  employed. 

Defining  a  methodology  rerpiires  choosing  a  software  development  model.  If  the  soft¬ 
ware  development  process  n<'<*ds  to  be  tailored  to  the  needs  of  each  .AI  ap|)liration. 
then  providing  a  meta-level  model  that  describes  how  to  define  a  software  develop¬ 
ment  process  appropriate  for  particular  .AI  project  will  prove  more  useful  than  "yet 
another  software  development  model.”  The  four  dimensional  model  described  below 
represents  an  attempt  to  provide  such  a  meta-level  model. 

4.1  Software  Life  Cycle  Models 

The  traditional  software  life  cycle  model  developed  initially  in  [36]  is  a  one  dimen¬ 
sional  model  that  portrays  all  software  development  projects  as  passing  through  the 
same  linear  progression  of  phases.  Over  the  years  there  have  been  many  different 
elaborations  of  this  linear  model — each  is  usually  proposed  as  universally  applicable 
to  all  software.  .More  recently,  a  two  dimensional  spiral  development  methodology 
has  been  proposed  by  Boehm  [6]  and  is  gaining  widespread  acceptance  because  it  ex¬ 
plicitly  recognizes  that  rapid  prototyping  is  often  useful  before  attempting  to  define 
the  requirements  for  the  proposed  system. 

In  this  report  we  propose  a  meta-level  model  of  alternative  software  development 
processes.  This  meta-level  model  is  especially  appropriate  for  .AI  software  where  there 
are  strong  concerns  about  verification  and  validation.  In  this  generalized  tnoilel.  whi<  h 
is  then  specialized  to  fit  the  needs  of  any  specific  software  project,  software  products 
(such  as  requirements  specifications,  performance  prototypes,  or  a  regression  test)  are 
represented  in  a  four  dimensional  space.  The  advantages  of  this  model  are: 

1.  Rather  than  providing  a  single  model  as  a  template  for  all  software  projects,  our 
meta-level  model  describes  how  to  define  a  development  process  tailored  to  the 
specific  needs  of  an  application — one  that  still  meets  V&V  needs.  .Any  software 
development  model  needs  to  be  adapted  to  fit  the  needs  of  a  specific  [uoject. 
hut  most  give  little  or  no  information  about  what  adaptations  are  valid. 


2.  It  focuses  on  the  products  of  software  development  more  than  on  the  develop¬ 
ment  process.  .Across  different  software  applications  there  is  more  commonalil  v 
in  the  final  products  than  there  is  in  the  seciuencing  used  during  the  development 
process. 

Our  met  a- level  model  encourages  the  definition  of  specific  software  deveio|)ment  plans 
that  produce  not  only  the  products  defined  by  existing  software  development  stan¬ 
dards  such  as  D0D-STD-2167A  but  also  the  additional  products  associated  with  rapid 
[)rototyping  and/or  formal  specifications.  D0D-STD-2167A  explicitly  allows  rapid  pro¬ 
totyping  methods  to  be  used  during  software  dev'elopment  but  does  not  provide  guide¬ 
lines  or  a  framework  for  when  and  how  to  use  rapid  prototyping  and  what  should  result 
from  different  forms  of  rapid  prototyping  efforts. 

Both  the  conventional  software  development  methodology  and  the  spiral  model  are 
'"ommon  special  cases  that  can  be  constructed  using  our  meta-levc!  model.  Since 
many  .AI  systems  are  embedded  in  applications  with  conventional  software,  it  is  im¬ 
portant  that  the  model  for  developing  and  validating  .AI  software  l)e  a  generalization 
of  conventional  development  methods. 

Our  general  model  recognizes  that  it  is  often  best  to  prototype  the  hardest  aspects  of 
the  problem — the  prototype  may  deal  with  functionality,  with  the  user  interface,  or 
with  the  execution  time  of  key  functions.  In  other  applications,  formal  specifications 
or  other  intermediate  software  products  are  useful  as  developmental  waypoints  while 
working  toward  the  final  software  products. 

DQD-STD-2167A  specifies  that  a  software  project  should  develop  and  use  its  own  in¬ 
dividualized  Software  Development  Plan  and  a  Software  Test  Plan.  Our  software 
development  model  is  intended  to  provide  a  framework  and  guidelines  that  .AI  soft¬ 
ware  developers  can  use  to  develop  these  plans  in  a  way  that  is  tailored  to  the  needs  of 
the  project  and  is  compatible  with  .AI  development  methods  being  used.  The  parts  of 
the  test  plan  that  deal  with  verifying  the  software  can  be  tailored  both  to  the  specific 
needs  ot  the  project  and  to  the  intermerliate  software  products  that  are  appropriate 
for  this  project. 

The  problem  with  a  software  development  model  that  is  uniform  for  all  software 
projects  is  that  either  it  constrains  the  <leveiopment  plan  to  fit  into  the  constraints 
of  a  single,  pre-conceived  developmental  process  or  else  it  doesn't  model  the  software 
development  process  in  very  much  detail.  For  example,  it  is  widely  recognized  that 
the  waterfall  model  should  be  adapted  to  the  needs  of  an  individual  project.  But 
the  waterfall  model  does  not  go  on  to  provide  a  framework  for  that  adaptation.  Our 
meta-level  model  provides  a  framework  for  constructing  more  detailed  development 
models  and  V&:V  plans  that  are  tailored  to  the  specific  needs  of  the  application. 
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4.2  The  Four  Dimensional  Software  Development  Model 


Suit  ware  development  begins  from  a  vague,  partial,  and  ambiguous  understanding 
of  file  [)roblem  to  be  solved,  and  it  progresses  toward  a  set  of  products  that  define 
an  unambiguous,  e.xecutable,  and  efficient  representation  of  a  problem  solution.  The 
evolution,  '.erificatiou,  and  validation  of  these  software  products  can  be  represent(>d 
by  paths  through  a  four  dimensional  space  where  the  dimensions  are  labeled  and 
scaled  as  follows: 


Definition: 

Formalization: 

Operationality: 

Efficiency: 


Partial  understanding 
of  problem 

.Ambiguous 

.Abstract 

Inefficient 


(.'omplete  understanding 
of  problem 

(.'nambiguous 

E.xecutable 

Efficient 


rhese  four  dimensions  correspond  closely  with  what  Hoare  characterizes  as  the  four 
contponents  of  a  complete  theory  of  programming. 


.\  complete  theory  of  programming  includes 

1.  .A  method  for  specification  of  programs  which  permits  individual  re¬ 
quirements  to  be  clearly  stated  and  coml)ined. 

2.  .A  method  for  reasoning  about  specifications,  which  aids  in  elucidation 
and  evaluation  of  alternative  designs. 

■].  .A  method  of  developing  programs  together  with  a  proof  that  they  meet 
their  specification. 

1.  .A  method  of  transforming  programs  to  achieve  high  efficiency  on  the 
machines  available  for  their  e.xecution. 

(From  Hoare  s  "Foreword"  in  [12!.  p.  Hi.) 


There  are  strong  interactions  between  these  four  dimensions  so  it  is  often  dilficult 
to  address  them  individually:  however,  software  complexity  can  be  reduced  and  \er- 
iftcation  and  validation  can  be  more  effective  when  there  are  software  products  that 
address  each  of  these  four  dimensions  separately. 

Logically,  it  may  seem  possible  for  software  development  to  proceed  sequentially 
through  each  of  these  four  dimensions.  Unfortunately,  there  are  strong  backward 
dependencies  among  these  dimensions:  that  is.  progress  in  one  dimension  often  cle- 
pends  on  prior  progress  in  a  later  dimension.  For  example,  before  defining  all  the 
requirements  for  a  software  system,  one  may  need  a  prototype  implementation  in 
(U'der  to  understand  what  is  practical  and  feasible  as  requirements.  Similarly,  with 
current  software  techniques,  it  is  seldom  practical  to  defer  performance  considera¬ 
tions  until  after  a  formal  specification  and  an  initial  operational  program  have  been 
<  om[)leted. 


These  backward  dependencies  among  the  four  dimensions  are  the  primary  reasons 
why  each  software  project  needs  to  have  its  own  Software  Development  Plan.  Thes<' 
backward  dependencies  are  different  for  rlifferent  software  applications  and  develop¬ 
ment  environments.  .A  Software  Development  Plan  should  aildress  these  l^ackward 
dependencies  by  working  out  the  relative  order  in  which  each  component  of  the  sys¬ 
tem  will  make  progress  in  each  dimension.  Our  lour  dimensional,  meta-level  model 
applies  both  to  the  software  system  as  a  whole,  to  each  of  its  compoiK'nts.  to  their 
components,  etc.  The  effort  to  divide  a  software  system  into  relatively  independent 
components  is  one  of  the  main  tasks  of  software  design,  and  it  is  no*  vlealt  with  by  t  h<" 
four  dimensional  model.  This  model  focu.ses  on  the  different  kinds  infori'ir.tion  to 
be  recorded  about  either  the  whole  software  or  about  any  component.  During  devel¬ 
opment.  it  should  be  expected  that  different  components  of  the  overall  software  will 
have  been  developed  to  different  stages  at  any  one  time.  .A  detailed  developmeiii  plan 
for  a  software  project  will  identify  the  software  components  and  the  relatixe  order  in 
which  each  component  will  progress  through  the  four  dimensions. 

The  four  dimensions  are  discussed  in  the  following  paragraphs. 


Definition,  This  dimension  1;  me  focus  of  a  requirements  specification.  When  a 
software  project  is  originally  conceived,  one  has  only  a  vague  uiidersi aiidiiig  of 
the  problems  to  be  s  ved  by  automated  processing  and  of  the  functions  iieeclerl 
to  solve  th  in.  The  definition  dimension  -measures  the  degree  to  which  the 
functional  .".id  performance  requtrerr  "“nts  of  ohe  system  have  been  iiuderstoorl 
anti  defined  in  some  fotm. 

Formalization.  This  dimmsion  measures  the  degree  to  which  the  software  is  de- 
Hned  ii  oine  formal,  precise,  and  unambiguous  notation  that  supports  deduc¬ 
tive  reasoning.  Formalization  removes  the  ambiguity  that  may  he  present  in  a 
requirements  specification.  More  important,  formalization  facilitates  deduction 
of  properties  about  the  system;  thus  enabling  properties  that  are  implied  by  a 
specification  to  be  made  explicit. 

Operationality.  .A  system  can  be  defined  (informally  or  formally)  at  an  abstrac  t 
level  without  mapping  this  definition  into  constructs  that  are  executable  on 
available  computer  hardware.  This  dimension  me.".'''rps  the  degree  to  which 
the  system  functionality  is  not  only  defined  but  also  executable.  Progress  lu 
this  dimension  can  be  thought  of  as  mapping  a  system  down  to  lower  levels  of 
abstraction — from  application-oriented  concepts  through  computer  abstractions 
like  queues,  stacks,  and  processes  and  on  down  to  bits  and  bytes. 

Efficiency.  This  dimension  deals  with  all  of  the  characteristics  of  the  system  that 
are  separate  from  the  input-output  functionality;  e.xecution  time,  response  time, 
memory  and  other  resource  utilization,  etc.  Programmers  usually  deal  with  both 
operationality  and  efficiency  concerns  together  during  the  implementation  pro- 
ess:  however,  operationality  and  efficiency  are  sometimes  separable  concerns. 
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Definition 


Figure  4-1;  The  Four  Dimensional  Space  for  Software  Development 

For  example,  functional  prototypes  can  be  distinguished  from  performance  pro¬ 
totypes. 


4.3  Evolving  Software  through  the  Four  Dimensions 

L  nlike  the  waterfall  models  of  software  development,  this  four  dimensional  represen¬ 
tation  does  not  itself  describe  stages  in  a  software  development;  rather  the  product 
of  a  development  stage  is  represented  by  a  point  within  this  four  ilimeiisional  space. 
( More  precisely,  a  development  stage  usually  results  in  several  products  that  are  rep¬ 
resented  by  a  set  of  points  together  with  mappings  between  those  points.) 

Figure  4-1  shows  a  representation  of  the  four  dimensional  space  for  software  devel¬ 
opment.  The  definition  dimension  is  along  the  vertical  axis,  operationaliiy  along  the 
horizontal  axis,  and  efficiency  is  represented  on  the  axis  coming  forward.  The  other 
axis  going  to  the  lower  left  in  the  fourth  dimension  represents  the  formalization  di¬ 
mension.  Since  it  is  difficult  to  visualize  a  four  dimensional  space,  our  figures  usually 
will  not  represent  this  dimension. 

The  need  for  documentation  to  justify  that  the  program  meets  its  goals  and  to  sup¬ 
port  maintenance  and  future  evolution  can  be  satisfied  by  products  represented  by 
points  in  the  four  dimensional  space  or  by  mappings  between  these  diverse  products. 
.-\  minimal  set  of  software  prorlucts  as  specified  by  D0D-STD-2167A  would  be  a  .Sys- 
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Figure  4-2:  The  Minitnal  Products  of  Software  Development 

tern  Requirements  Specification,  a  Software  Design  Document,  the  Source  (.'ode  and 
Listings  of  the  completed  program  and  the  Software  Test  Reports. 

These  products  are  represented  in  the  four  dimensional  space  as  shown  in 
Figure  4-2.  The  .Software  Test  Reports  are  represented  not  by  a  point  in  the  space  but 
by  the  mapping  from  the  comi)leted  program  to  the  requirements.  (Other  documents 
required  l)y  2167A  are  a  Software  Development  Plan  and  a  Software  Test  Plan.  The 
Software  Development  Plan  defines  a  path  that  the  project  will  follow  through  the 
four  dimensional  space.  The  four  dimensional  representation  is  an  aid  for  designing 
the  Software  Test  Plan,  but  the  plan  is  not  represented  in  the  four  dimensional  space. 
The  Operator's  and  Laser's  .Manual  are  also  required  products  but  are  orthogonal  to 
the  issues  described  by  the  four  dimen.sional  model.) 

Our  generalized  model  allows  a  wide  variety  of  other  possible  intermediate  software 
products  to  be  defined  as  [joints  in  the  fom  dimensional  space.  These  additional 
software  products  can  be  used  as  additional  waypoints  during  software  development; 
thus  the  standard  products  required  Ijy  2167A  are  particular  cases  of  the  software 
products  that  can  lie  represented  using  the  four  dimensional  model. 

Waterfall  models  specify  a  fixed  sequential  process  for  generating  these  software  pro<l- 
ucts.  In  our  four  dimensional  nio<lel,  we  represent  the  software  products  that  result 
from  a  waterfall  model,  but  we  do  not  define  a  fixed,  sequential  development  process; 
and  we  make  it  easier  to  include  additional  products  as  waypoints  during  software 
development. 
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Figure  4-3:  The  Spiral  Development  Model  Represented  within  the 
Four  Dimensional  Space 

4.4  Rapid  Prototyping  and  the  Spiral  Development  Model 

Boehm  [6]  describes  the  software  development  process  as  a  spiral.  Each  cycle  of 
the  spiral  is  devoted  to  resolving  the  highest  risk  issues  involved  in  this  particular 
application.  Experience  from  the  implementation  and  use  of  each  prototype  is  n.«ed 
to  develop  a  more  complete  definition  of  the  functionality  and  performance  that  is 
desired  in  the  final  product.  Figure  4-3  shows  how  the  products  of  a  typical  -^i^iral 
development  would  be  represented  within  the  four  dimensional  model.  One  liegms 
by  prototyping  and  defining  the  concept  of  operation  for  the  application.  Then  one 
prototypes  and  defines  the  .software  requirements,  and  finally  one  builds  a  strucrural 
prototype  leading  to  the  definition  of  the  software  design.  The  completed  program  tor 
the  system  that  is  to  become  operational  is  then  implemented  based  on  this  design. 
.\n  advantage  of  the  four  dimensional  model  is  that  it  provides  a  framework  for 
thinking  about  alternative  development  processes  that  use  alternative  waypoints  in 
order  to  arrive  at  and  verify  the  final  products  of  the  software  development  process. 
Rec|uirements  documents,  design  documents,  and  various  forms  of  prototypes  can  <dl 
be  understood  as  intermediate  software  products  associated  with  particular  points  m 
the  four  dimensional  space. 
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Formalization 
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Figure  4-4;  Different  Kinds  of  Prototype  Implementations 

With  respect  to  prototypes,  it  is  important  to  specify  what  functionality  or  efficiency 
characteristics  one  is  trying  to  prototype.  The  implementation  rtfort  involved  in 
Iniilding  a  prototype  will  be  large  if  one  does  not  carefully  focus  the  prototyping 
activity  on  critical  issues.  Figure  4-4  shows  five  different  kinds  of  |)iorotypes  located 
at  different  points  in  the  four  dimensional  space. 


Functional  Prototype.  This  is  an  implementation  of  some  critical  functionality  to 
test  whether  it  can  be  implemented  successfully.  For  example,  if  a  large  system 
is  being  built  that  includes  automatic  recognition  of  targets  from  certain  kinds 
of  images  or  signals  and  if  similar  functionality  has  not  been  flemonst rated  in 
previous  systems,  then  a  functional  prototype  to  lest  the  feasibility  of  achieving 
this  functionality  is  indicated. 

User  Interface  Prototype.  If  the  interaction  between  the  systfuti  and  the  user  is  to 
be  quite  different  from  that  of  any  previous  system,  then  a  prototype  that  rloes 
not  have  much  internal  functionality  but  simulates  the  proposed  user  interface 
and  evaluates  it  in  actual  tests  with  representative  users  is  indicated. 

Performance  Prototype.  If  some  of  the  functions  required  in  a  proposed  system 
have  never  been  implemented  with  adequate  response  times  using  the  proposed 
processing  resources,  then  a  prototype  to  test  these  functions  running  on  the 
proposed  hardware  (or  a  simulation  of  it)  is  indicated. 


Design  Prototype.  A  design  or  structural  prototype  is  used  to  evaluate  the  effec¬ 
tiveness  of  a  proposed  software  design.  It  typically  addresses  a  comlunatioii  of 
critical  functional  and  performance  issues. 

Operational  Prototype.  Sometimes  it  is  useful  to  test  the  majority  of  the  function¬ 
ality  of  the  proposed  system  running  in  an  environment  similar  to  the  ultimate 
operational  setting.  .As  represented  in  the  four  dimensional  space,  an  opera¬ 
tional  prototype  may  involve  much  of  the  e.xpense  of  a  completed  prodiu  t.  aiul 
it  is  usually  useful  to  plan  it  so  that  the  majority  of  it  can  be  evolved  for  use 
in  the  final  product. 

One  advantage  of  the  four  dimensional  model  is  that  the  differences  between  dilfereni 
kinds  of  prototypes  are  explicitly  represented  in  the  model. 


4.5  Formal  Specifications  as  Software  Development  Waypoints 

Different  kinds  of  prototypes  are  not  the  only  additional  software  development  way- 
points  that  are  represented  with  the  four  dimensional  model.  By  considering  the 
dimension  dealing  with  formalization,  different  forms  of  specification  can  also  be  a.s- 
sociated  with  points  in  the  four  dimensional  space.  These  various  specification  options 
are  important  for  \‘kV.  Figure  4-5  shows  some  of  these  specification  options  as  points 
in  the  four  dimensional  space. 


) 


Oafinitien 


Figure  4-5:  Additional  Possible  Specification  Waypoints 

4.6  V&V  as  Mappings  between  Software  Products 

In  our  meta-level  software  development  model,  verification  and  validation  are  nor 
represented  as  points  in  the  four  dimensional  space;  rather,  they  are  represented 
as  mappings  l^etween  points.  Both  verification  and  validation  are  concerned  with 
showing  that  the  implemented  software  satisfies  the  software  requirements.  V  alidation 
does  this  directly  by  testing  and  other  methods  that  directly  evaluate  the  software 
implernentation  against  requirements.  In  Figure  4-6.  validation  is  shown  as  a  direct 
mapping  between  the  implemented  code  and  the  software  requirements  flocument. 

Verification  uses  an  indirect  path  through  intermediate  software  products  to  show 
that  the  final  software  meets  the  requirements.  The  indirect  path  that  is  used  in 
conventional  software  <levelopment  is  shown  in  Figure  4-7.  We  propose  that  the 
specific  choice  of  the  intermediate  waypoints  for  verification  are  less  important  than 
the  beisic  process  of  showing  that  the  final  e.xecutable  software  maps  through  some 
series  of  intermediate  waypoints  back  to  the  system  requirements. 

The  reason  for  doing  verification  using  intermediate  waypoints  is  that  the  mapping 
from  one  waypoint  to  the  ne.xt  can  be  much  simpler  than  the  mapping  all  the  way  from 
the  final  software  back  to  the  requirements:  and  additional  techniques  can  be  applied 
to  verify  that  the  software  product  at  one  waypoint  satisfies  the  requirements  imposed 
by  the  previous  waypoint  when  the  conceptual  gap  between  the  two  waypoints  is 
relativelv  small. 
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Figure  4-6:  Validation  as  a  Mapping  between  the  Implementation 
and  Requirements 


Sometimes  software  verification  is  viewed  as  showing  that  a  correct  software  devel¬ 
opment  process  has  been  followed.  We  prefer,  however,  to  emphasize  verification 
as  focused  on  showing  that  the  content  of  the  products  of  eacli  stage  of  software 
devf'lopment  satisfy  the  requirements  for  those  products. 

Once  one  introduces  prototype  development  as  a  legitimate  part  of  the  software  de- 
velo[)m(*nt  process,  one  does  not  want  to  require  that  the  waypoints  used  to  verify 
soltware  have  to  correspond  completely  with  all  of  the  developmental  waypoints  used 
during  the  evolution  of  the  software.  .A  prototype  may  be  used  largely  in  order  to 
define  an  appropriate  requirements  specification.  Thus  various  prototype  software 
products  may  be  part  of  the  software  development  history  but  they  need  not  iiie  used 
as  intermediate  waypoints  in  the  verification  plan. 

The  reason  why  software  verification  is  needed  in  addition  to  software  validation  is 
that  .AI  software  implementations  are  so  comple.x  that  testing  the  software  directly 
against  re(|uirements  cannot  be  complete.  .Software  complexity  also  makes  verification 
difficult:  however,  there  are  techniques  for  reducing  the  complexity  of  the  mappings 
Irom  implementetl  software  through  intermediate  waypoints  to  a.  requirements  speci- 
lication.  .As  discus.sed  in  [27,  29],  examples  of  these  techniques  include: 


•  Automate  parts  of  the  evolution  from  the  requirements  specification  to  the 
implementation,  (.'ompilers  for  high  level  programming  languages  automate 
part  of  the  work  in  the  <lirection  of  operationality.  and  automatic  programming 
lorhniques  that  are  ( apable  of  executing  and  optimizing  specifications  writ  ten  in 
very  high  level  languages  can  contribute  further  automation.  It  is  still  nei  cssarv 
to  verify  that  the  automatic  compilation  preserves  correctness  hut  automat  ion 
means  that  this  verification  effort  need  not  be  done  repeatedly. 

•  I'se  software  engineering  techniques  to  constrain  the  design  and  the  imple¬ 
mentation  so  that  the  mapping  back  to  the  requirements  specification  is  more 
understandable  and  h'ss  complex. 

•  Tocus  on  verifying  the  most  critical  functionality  in  the  requirement  s|)e<  ifica- 
lion. 

•  .Make  I  he  sat  isfact  ion  of  cril  ical  requirements  depetid  on  only  small  [larts  ot  t  he 
implementation. 

•  .Separate  control  and  efficiency  is.sues  from  the  functionality  of  the  imph'menta- 
tion. 

•  .Make  the  implementation  be  partially  self-checking. 

4.7  Work  Remaining  on  the  Meta-leveJ  Development  Model 

More  work  remains  to  elaborate  this  four-dimensional  software  development  moilei 
before  it  will  be  useful  as  a  guideline  for  developing  and  verifying  .A1  software.  In 
particular,  we  need  to  develop  and  test: 

•  .Metrics  for  measuring  progress  in  each  of  the  four  dimensions. 

•  Definitions  for  additional  waypoints  that  are  useful  during  software  development 
and  verification. 

•  Examples  of  how  different  sets  of  waypoints  can  be  used  successfully  in  dilFiueni 
kinds  of  .\I  software  development  projects. 

•  .Additional  methods  for  doing  verification  by  mapping  between  these  waypoints. 

Once  these  waypoints  and  metrics  are  defined,  a  software  development  and  verification 
plan  can  be  formulated  as  a  set  of  waypoints  or  stepping  stones  that  are  located 
within  this  four  dimensional  space.  Planning  a  software  development  process  is  then 
analogous  to  identifying  stepping  stones  for  crossing  a  river.  One  tries  to  find  .> 
series  of  stepping  stones  that  are  close  enough  so  that  software  development  (  an 
proceed  in  a  series  of  relatively  small  conceptual  steps  with  each  step  l)eing  relia!>l(' 
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and  verifiable.  One  does  not  always  have  to  use  the  same  stepping  stones,  one  just 
has  to  find  stepping  stones  that  are  close  enough  together  so  the  complexity  of  rhe 
conceptual  gap  between  them  allows  verification  techniques  to  be  applied  successfully. 
If  we  think  of  the  comple.xity  of  the  complete  software  system  as  analogous  to  the 
width  of  the  river,  then  this  analogy  indicates  that  the  number  of  waypoints  that  are 
nec'ded  in  a  software  development  plan  is  not  the  same  for  all  software  projec  ts  but 
is  pioportionate  to  the  complexity  of  the  software. 

4.8  Conclusions  about  the  Meta-level  Software  Development  Model 

Each  software  project  needs  to  define  the  specific  software  products  that  arc  to  Ije 
produced  by  the  project  and  the  order  in  which  they  should  l)e  gen('rati'd.  Our  haii' 
dimensional  meta-level  model  provides  a  way  to  represent  both  the  final  piodiu  ts. 
the  intt'rmediate  (nodncts  that  are  used  as  way-points  in  working  lowarrl  the  lintd 
products,  and  the  validation  and  verification  mappings  between  them. 

The  four  <limensional  mode!  is  useful  in  laying  out  a  .Software  Development  Plan  that 
is  tailored  to  the  needs  of  the  specific  project.  Certain  software  products  will  appear 
in  the  plan  tor  every  project  —  tor  example,  requirements  specificai  ions  and  tlesign 
specifications.  The  level  o‘'  detail  that  is  achievable  in  these  products  may.  however, 
differ  with  different  jects.  Prototypes  and  formal  specifications,  however,  need  to 
be  planned  to  me  ^ecific  needs  of  each  project.  The  verification  and  validation  plan 
may  also  vary  -.viin  specific  project  needs.  Prototypes  that  are  fleveloped  primarilv 
to  understand  the  application  requirements  will  probably  not  jtlay  a  major  role  in 
the  \'L\  plan:  however,  an  executable  specification  may  serve  as  a  prototyi^e  and 
also  be  used  as  an  intermediate  waypoint  in  verification  by  showing  that  the  final 
optimized  code  is  equivalent  in  functionality  to  the  executable  specification  which 
itself  satisfies  the  requirements.  VVe  believe  that  this  four  dimensional  model  for 
software  development  will  provide  the  flexibility  and  adaptability  needed  to  tailor  .\I 
software  development  to  the  needs  of  the  application  while  also  providing  a  framework 
with  enough  structure  so  that  verification,  validation,  and  maintenance  of  the  software 
can  be  effective. 
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Part  II 


HOW  TO  SUPPORT  V&V 


5.  For  Technical  Leaders:  Planning  to  Support 


This  section  is  devoted  to  piovidiiig  technical  leaders  on  AI  projects  with  some  guid¬ 
ance  on  how  to  support  WtV  during  software  project  planning.  It  touches  on  a 
number  of  issues  that  are  addressed  more  fully  in  subsequent  sections,  as  well  as  on 
issues  specific  to  planning. 

5.1  What  Level  of  V&V  is  Appropriate? 

Perhaps  the  most  fundamental  e.xample  of  a  planning-specific  issue  is  how  to 

determine  the  level  of  \  iS.:V  that  is  appropriate  for  the  project.  lietter  verification  and 
more  comprehensive  validation  can  always  be  obtained  by  devoting  more  resourc('s 
to  the  process.  But  resources  are  limited,  and  some  of  those  resourci's  must  h<> 

ilevoted  to  other  tasks.  Before  the  iletailed  tradeoffs  among  potential  procedures 

are  determined,  the  total  level  of  effort  that  will  be  devoted  to  VSeV  should  be  settled. 
While  both  the  level  of  V',?cV  and  the  particular  procedures  to  be  applied  are  often 
subject  to  external  constraints — organizational  policies  or  contr.actual  requirements, 
for  example — that  somewhat  simplify  the  problem,  a  careful  analysis  of  the  "naturar’ 
requirements  on  the  project  is  still  essential  to  making  the  best  use  of  scarce 
development  resources. 

Guideline  5.1  Ddermtnt  the  total  level  of  effort  to  be  devoted  to  I  T'’  !  ’,  based  on  the 
charnctenstics  of  the  project,  at  the  beginning. 


5.2  Project  Categorization:  An  Example 

One  generally  useful  technique  for  determining  the  level  of  \  for  a  project  is  to 
use  a  categorization,  based  on  salient  characteristics  of  the  project,  to  determine 
what  sorts  of  V&V  activities  are  necessary  and  appropriate.  The  details  of  such  a 
categorization  should  be  determined  by  the  sorts  of  projects  that  your  organization 
performs.  Below  you  will  find  an  example,  based  on  a  categorization  found  useful 
rit  the  organization  where  the  authors  are  employed,  that  should  provide  a  usefid 
l)aseline  for  your  category  definition  efforts. 

First,  here  are  the  criteria  u.sed  in  rletermining  the  category  of  a  project.  Each 
criterion  is  rated  as  being  satisfied  at  a  Low  level,  a  Medium  level,  or  a  High  level  on 
the  given  project.  Table  5-1  explains  what  the  three  ratings  mean  for  each  criterion. 
(.Vole  that  some  of  the  criteria  are  marked  with  an  asterisk.  '*’.  The  significance  of 
this  will  be  explained  in  the  definition  of  the  categories.) 

Our  three  categories  of  projects  are  simply  called  .\.  B.  and  C.  Which  category  a 
project  belongs  to  is  determined  from  its  ratings,  as  follows. 
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Criterion 

Rating=Low 

Rating=Medium 

Rating=High  ] 

*  I’mt  (if  system 

Developer  only 

.Vver-me  folks 

('out rati  Typ** 

CPFF, 

FPLOE,  CPAF. 
rPIF 

*  Fli'xihiliiy  in 

D'  iulliiie  for 

Dilivcrv 

No  requirement 

Negotiable 

Fi.xed  .Hid  ! 

iion-iiegotialdc' 

!  D*‘li\('ry 

!  Eiivuomuent  (H/W 

ami  (3/S) 

Exotic  (e.g.. 
('oiinection 

Machine) 

Unix  workstation 

P(  '  imi Tf 'proi  essor 

hoai'ii 

1  (roupliiig  to 

Exismh"  Svstems 

Standalone 

workstation 

Loosely  coupled 

i  ; 

*  Execution  Speed 

1  Requirements 

Easily  met 

Difficult  to  achieve 
speed  perceived  as 
satisfactorv 

Hard  re.al  iime 

reipiin  nients  must 

be  met  ! 

Design  Formalism 

! 

Developer  s  choice, 
designs  not  subject 
to  customer  review 

Designs  subject  to 
review,  familiar 
design  formalism 

( 'iistoiiier-deriiied 
unfamiliar  design 
formalism 

Darahase/ Knovv- 
leilne  Base 

DBMS/KBMS  of 
developer's  choice, 
data  already 
recorded,  knowledge  i 
already  formalized 

Developer’s  choice, 
but  substantial  data 
acquisition/ 
knowledge 
engineering  etfori 
required 

.Mu.si  interface  to 
( '0  I  S  'oft  ware  of 
customer '.s  rlioice. 

1  substantial  elfort 
ilevoli'd  to 
i  poinilanng  DB  / KB 

Doeumeiitation 

Simple  User's 

Manual  only 

User's  Manual. 
Maintenance 

Vf;^niial 

Full  JlhT.V  : 

*  I'esrmg 

Rei|iiiremeiUS 

Demo  only 

I 

Testing  at  system 
level  must  be 

ilnruniented 

Unit  testing  must 
be  documented 

(  oiifimiration 

Management 

Requirements 

None 

j 

Manual  C.M 

.Aniomateil  (  '.M  tool 
employed 

Support  Software  to 
he  Developed 

j 

None 

On-line  help  system 

!  On-liiii'  help 

1  utonal,  systi'in 
status  monitoring. 

etc _ I 

Table  o-l:  Criteria  Used  in  CategOiizing  Projects 


•  If  a  project  rates  a  High  on  any  of  the  asterisk-marked  criteria,  its  category  is 
A. 

•  If  a  project  rales  a  maxinuiin  of  Mediiiiii  on  th('  asterisk- markerl  criteria,  then 

0  if  it  rates  a  High  on  the  majority  of  the  iuunarke<l  criteria,  its  <  ategory  is  A. 
o  else  its  category  is  B 

•  If  a  project  rates  a  Low  on  all  asterisk- marked  categories,  tlien 

0  if  it  rates  at  least  Medium  on  the  majority  of  the  unmarked  criteria,  its 
category  is  B. 

o  else  its  category  is  ('. 

Lach  cat('gor\'  has  an  assoc  iaied  baseline  level  of  documentation  that  must  he  pro¬ 
duced  and  procedures  that  must  be  followed  to  support  \1^\’  on  the  [noject.  i  See 
Appendi.x  B  for  definitions  ot  acronyms.) 

Categorv  A  projects  must 

o  produce  a  Software  Development  Plan  prior  to  development. 

0  produce  an  SRS.  IRS.  SDD.  IDD,  STP.  STD.  and  STR  during  development. 
0  define  and  follow  formal  configuration  management  procedures. 

0  appoint  a  ciualified  System  Engineer  as  Program  Technical  Director. 

0  institute  a  formal  Quality  .Assurance  program. 

0  perform  formal  testing, 

0  employ  a  formal  design  method. 

0  publish  a  detailed  schedule. 

0  institute  a  process  for  collecting  and  handling  SER's. 

0  conduct  formal  Engineering  Reviews  (design  reviews,  code  walkthroughs, 
etc. ),  and 

0  participate  in  the  company  s  TQ.M  program  by  conducting  ((uarterly  Program 
Reviews. 

B.  Category  B  projects  must 

0  produce  a  Software  Development  Plan  prior  to  development, 

0  produce  an  SRS.  SDD.  and  STP  during  development. 

0  define  and  follow  configuration  management  procedures. 

0  appoint  a  Program  Technical  Director, 
o  publish  a  schorlule. 


0  conduct  (possibly  iiifornial)  Engiiiet^ring  Reviews,  and 

0  part  icipate  in  the  company'-;  TQM  program  by  conducting  quarterly  Prograin 
Reviews. 

('.  Category  ('  projects  must 

0  dehne  high-level  goals  atid  plans  for  the  project. 

0  publish  a  scheduh'  of  major  milestones,  and 

0  participf  te  in  the  company's  TQM  program  by  conducting  quarterly  Program 
Reviews. 


This  sort  of  categorization  provides  the  level  of  \'S^V  appropriate  to  the  |)roJect.  and 
allows  at  least  a  rough  prediction  of  the  percentage  of  project  resources  that  must  be 
devoted  to  V’lCV  .  while  lea\  iiig  i  he  procedural  details  open. 


6.  For  System  Specifiers:  What  and  How  to  Specify 


6.1  The  Role  of  Requirements  Analysis  in  AI  Development 

Re(|uirenients  are  properties  a  system  must  have  iti  order  to  solve  the  prol^lem.  The 
hulk  of  a  requirements  definition  document  will  often  he  a  statement  of  the  proh- 
leni.  Indeed,  the  most  difficult  part  of  requiremetits  analysis  is  typically  achieving  a 
suiru  iently  detailed  understanding  of  the  problem.  However,  leaving  the  ac  tual  state¬ 
ment  of  requirements  at  tlie  level  of  “the  only  requirement  is  to  solve  the  |)roblem"  is 
usually  unsatisfactory  because  of  vagueness  in  the  problem  statement.  Re([iiiremenrs 
should  be  defined  sufficiently  precisely  that  a.  given  system  either  satisfies  them  or 
does  not  satisfy  them.  .Moreover,  it  must  be  possible  to  determine — with  a  degree  of 
rertainty  that  is  itself  a  re(|uirement  on  the  system — whether  a  system  satisfies  the 
re(|uirements  or  not.  (Certification  of  requirement  satisfaction  is  accomplished  via  a 
combination  of  verification — formal,  informal,  or  some  combination  of  the  two — and 
validation  testing.) 

Guideline  6.1  System  requirements  should  be  spelled  out  in  enough  detail  that  a 
system  either  satisfies  them  or  fails  to  satisfy  them:  do  not  leare  the.  n quiremen,s 
statement  "fuzzy." 

,\s  was  pointed  out  in  Section  2,  it  is  important  to  remember  that  the  product  of 
re.qairements  analysis  is  not  necessarily  the  true  requirements.  I’se  of  the  term  "re- 
((uireinents  definition"  is  appropriate  from  the  contractual  point  of  view — where  a 
system  will  he  accepted  if  and  only  if  it  satisfies  the  requirements  «•>  defined  in  the 
contract — but  somewhat  misleading.  It  is  quite  possible  that  a  system  satisfies  the 
re(|uirements  as  stated  yet  fails  to  solve  the  problem,  due  to  an  error  in  the  require¬ 
ments  analysis  or  a  loophole  in  the  requirements  definition.  Systems  validation  should 
he  jjerformed  against  the  true  requirements;  satisfaction  of  the  stated  requirements 
is  evidence  that  the  requirements  are  satisfied,  hut  it  is  not  conclusive  evidence.  The 
important  point  to  keep  in  mind  is  that  comprehensive  testing  against  the  stated 
requirements  cannot,  in  general,  provide  comprehensive  validation.  .\  requirements 
definition  document  is  just  another  lifecycle  product,  anil  no  amount  of  verification 
that  the  system  satisfies  the  stated  requirements  can  eliminate  the  need  for  validatioi\ 
testing. 

Guideline  6.2  Never  confuse  the  real  requirements  with  the  stated  requirements,  be- 
raiise  there  is  no  guarantee  that  the  requirements  hare  been  stated  correctly. 

common  characteristic  of  .A I  system  development  elforts.  as  noted  in  .Section  4.  is 
that  the  requirements  are  vague  and  hard  to  define.  I'liis  is  partly  due  to  the  nature  of 
the  problems  addre.s.sed  by  .A I.  .Just  as  it  is  hard  to  define  a  performance  measure  that 
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captures  how  well  an  expert  does  his  job,  it  is  hard  to  define  a  performance  measure 
that  captures  how  well  an  expert  system  performs  the  same  job.  It  is  also  partly 
d  e  to  the  fact  that  .\I  techniques  were  developed  to  address  large,  comparatively 
ill-defined  problems.  .A  good  example  is  search.  Conventional  techiiiriues  perform 
well  when  the  search  space  is  comparatively  small,  allowing  effectively  exhaustive 
exploration  of  all  alternatives,  and  there  are  well  defined  criteria  for  what  counts  as 
a  good  solution.  When  .AI  is  applied  to  search,  it  is  typically  because  the  search 
space  is  too  large  or  too  ill-behaved  for  conventional  technicines,  or  because  there  are 
no  effective  criteria  that  distinguish  good  solutions  from  others.  Often,  the  I)est  you 
can  do  in  such  cases  is  to  build  a  system  that  emulates  a  person  who  is  believed  to 
perform  his  job  effectively.  This  is  just  the  sort  of  ca.se  where  there  seems  to  be  little 
hope  of  accurately  defining  the  requirements. 

So.  does  this  mean  that  .AI  development  projects  should  not  waste  effort  on  attempt¬ 
ing  to  tieline  system  requirements?  The  typical  .\I  lifecycle.  ba.sed  on  iterative  refine¬ 
ment  of  a  prototype,  might  seem  to  support  an  affirmative  answer.  The  developer 
aslvs  the  user  whether  the  .system  is  satisfactorily  in  some  given  respect;  if  the  user 
is  tiot  satisfied,  he  is  asked  how  the  system  must  be  changed  so  that  he  will  be.  The 
end  product  of  this  refinement  process  is  the  prototype  that  embodies  the  rec|uire- 
ments.  rather  than  a  document  that  attempts  to  state  the  requirements.  What,  if 
anything,  is  wrong  with  this?  Doesn't  the  prototype  provide  even  better  guidance  in 
the  subsequent  evolution  of  that  system  than  written  requirements  would? 

The  ansu-er  to  that  last  question  is  a  resounding  “.No!’’  To  see  why.  we  must  take 
a  closer  look  at  the  role  of  requirements  statements  in  the  system  lifecycle.  Re¬ 
quirements  play  a  quite  different  role  in  the  lifecycle  than  specifications  and  designs. 
Requirements  are  determined  by  the  problem  itself,  and  cannot  be  changed  by  the 
implementors.  If  a  system  does  not  satisfy  the  requirements,  it  does  not  solve  the 
problem  f  although  it  may  solve  some  closely  related  problem).  Thus  the  requirements 
explicitly  present  the  constraints  on  the  design  process.  .\nd  this  is  the  key  to  seeing 
why  an  explicit  requirements  definition  is  essential. 

Consider,  for  concreteness,  a  mock-up  of  a  system's  u.ser  interface.  .\  u.ser-approved 
mock-up  of  the  user  interface  cannot  usually  l)e  considered  an  adecpiate  reciuirements 
definition  for  the  interface,  for  several  rea.sons.  First,  some  features  of  the  mock-up 
that  were  considered  artifacts*  by  its  creators  may  have  been  essential  in  securing 
the  user's  approval.  Perhaps  color  of  some  green  icon  was  selected  at  random  by  the 
developer,  yet  the  user  vvould  not  have  approved  the  interfaCv?  if  the  icon  had  been 
any  other  color.  Unless  the  essential  features  of  the  user  interface  are  distinguished 
from  the  accidental  features,  people  who  subsequently  want  to  change  the  interface, 
for  one  reason  or  another,  have  no  guidance  on  what  can  be  changed  and  what  must 

‘  Artifact,"  as  it  is  lieing  used  here,  is  .a  technical  term.  It  means  that  the  feature  is  the 
result  of  an  arbitrary  choice  made  by  the  developer  and  that  it  is  not  intended  to  be  a 
feature  that  the  evctual  u.ser  interface  mu.st  share  in  order  to  be  considered  essentially  the 
same  as  the  mock-up. 


remain  the  same.  Second,  “’tolerances”  are  left  implicit,  so  questions  Mich  as  "Is  it 
acceptable  to  make  this  window  10%  narrower?”  <  annot  l)e  answered. 

Thus,  while  prototy|)ing  is  a  useful — sometimes  virtually  essential — technique  tor  de- 
ti'rniiuiug  system  requirements,  the  prototype  alone  is  not  aii  adequate  ri'([uireiueuts 
dehuition.  Ideally,  the  prototype  would  be  supplemented  with  a  list  of  all  its  relevant 
features,  classifii'd  as  essential  or  accidental,  together  with  a  range  of  ac(  eptable  vari¬ 
ation  in  each  essential  feature.  This  ideal  may  not  be  achievable.  It  may  not  e\(>n 
be  ail  appropriate  goal  on  a  given  project.  The  point  is,  a  requirements  dehuition 
is  sui)pose  to  say  what  features  the  system  must  have,  and  this  informal  ion  ( aiinol 
be  extracted  from  the  prototype  alone.  Thus  a  prototype  does  not  prov  ide  the  same 
guidance  to  tlesigners.  implementors,  and  maintainers  that  a  reciuireiiu'uts  deliiiition 
does.  Far  from  reducing  the  value  of  writing  a  requirements  definition,  proioivping 
inci'Kises  it  by  helping  to  ensure  that  the  requirements  are  correct  and  sufHcicnilv 
detailed  that  they  will  provide  practical  guidance! 

.-\nd  the  problem  becomes  even  more  acute  when  tnodifications  are  made  to  a  'V  'tein 
that  has  already  been  validated.  As  a  simple  example,  consider  the  addition  of  a  new 
window,  which  will  display  a  new  sort  of  information  that  has  become  available  to 
the  user.  To  what  extent  can  other  windows  be  moved  or  resized  to  accommodate 
this  change?  I'he  original  requirements  on  their  size  and  position  prov  irle  an  excid- 
lent  starting  point  for  answering  the  question:  if  one  window  was  already  as  ^mall 
as  requirements  allowed,  while  another  was  far  larger  than  required,  then  shrinking 
the  latter  rather  than  the  fortner  will  almost  always  make  the  user  happier  than  the 
opposite.  Without  some  record  of  requirements  and  design  decisions  that  rletermined 
window  sizing,  there  is  no  basis  for  preferring  either  alternative  over  the  other.  While 
this  example  is  admittedly  trivial — the  modifier  can  ask  the  user  which  window  <hould 
l)e  shrunk,  or  can  even  get  a  mock-up  of  the  new  interface  approved — in  more  com¬ 
plicated  examples,  the  guidance  provided  by  the  requirements  would  be  even  more 
welcome. 

Guideline  6.3  Prototyping  helps  to  define  the  requirements,  hut  do  not  nttfinpt  to 
u.sf,  II  prototype  as  n  requirements  definition. 


6.2  The  Role  of  System  Specification  in  AI  Development 

.A  specification  is  a  description  of  the  system  to  be  built  at  a  “black  box'  level  of 
abstraction.  This  rlescription  should  be  declarative,  not  procedural,  in  that  it  <hould 
say  w/iat  the  system  does  without  prescribing  how  the  system  does  it. 

Thus,  requirements  and  specification  are  quite  distinct.  Requirements  deal  with  prop¬ 
erties  of  systems  in  general:  specifications  are  descriptions  of  particular  systems  (albeit 
at  a  very  abstract  level).  Every  system  that  solves  the  problem  must  satisfy  the  re- 
(piirements.  but  many  incompatible  system  specifications  are  possible.  The  problem 
completely  determines  the  requirements,  but  merely  constrains  the  specification.  .\ 
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requirements  document  says  what  the  system  must  do;  a  specification  says  u  h.it  the 
system  shnll  <lo.  In  fact,  QA  personnel  on  large  aerospace  development  efforts  (  he<  k 
that  the  right  word  gets  used:  statements  of  requirements  must  contain  the  woi<l 
■‘must,”  never  the  word  “shall;”  specifications  must  contain  "shaH”  and  never  ■‘must  .  ’ 
A  design  is  a  refinement  of  a  spec  ification:  conversely,  a  specification  is  a  tep-level 
design.  The  ”l)lacl\  box"  of  the  specification  is  broken  up  into  smaller  boxes  (more  for¬ 
mally,  abst raction  units),  those  boxes  are  broken  up  into  smaller  boxes  still,  ami  so  on. 
Eventually,  very  simple  boxes — i.e.,  ones  that  are  straightforward  to  implement —are 
arrived  at.  The  result  of  this  decomposition  is  t  he  detailed  design.  The  exact  nature 
of  the  boxes  depends  on  the  design  methodology  being  employed,  but.  in  many  <  a.ses. 
they  correspond  to  abstraction  mechanisms  in  the  implementation  language  to  be 
employed.  In  other  words,  design  is  simply  identification  and  specification  of  parts  of 
the  system.  Thus  a  general  purpose  specification  language  can  be  employed  through¬ 
out  the  design  process,  making  the  design  stages  appear  to  be  successive  relinement 
steps,  .\lternatively.  different  design  formalisms  can  be  employed  at  ditforenl  le\('ls 
of  abstraction.  Unfortunately,  the  latter  seems  to  be  standard  aerospiice  practice— at 
least  to  the  extent  of  using  natural  language  or  simple  diagrams  for  specification  and 
high  level  design  and  some  sort  of  pseudocode  for  detailed  design. 

This  account  is  at  odds  with  the  view  that  the  dividing  line  between  requirements  and 
specification  is  not  sharp,  that  requirements  simply  tend  to  be  more  domain-specific 
and  specifications  more  detailed  and  complete.  There  is  a  grain  of  truth  in  this  view, 
in  that  satisfaction  of  the  specification  should  imply  satisfaction  of  the  requirements. 
Hence,  there  is  a  sense  in  which  all  the  information  in  the  requirements  is  captured  in 
the  specification.  However,  the  view  that  a  specification  is  obtained  by  adding  detail 
to  the  requirements  definition  is  fundamentally  mistaken.  It  is  important  to  avoid  the 
error  of  conflating  the  two  because  requirements  and  specifications  play  very  different 
roles  during  the  maintenance  pha.se  of  the  lifecycle. 

Guideline  6.4  Do  not  confuse  specified  properties  and  requirements.  There  is  noth¬ 
ing  ivrong  with  a  sijstern  that  satisfies  the  requirements  bat  not  its  specification  i though 
the  specification  should  be  enentuallg  modified  to  fit). 


.\s  was  pointed  out  in  the  previous  section,  the  role  of  the  requirements  definitioti 
is  to  provide  constraints  on  the  rest  of  the  development  process.  This  tlocriment 
changes  only  if  an  error  is  discovered  in  the  reriuirements  analysis.  .A  specification  is. 
on  the  other  hand,  entirely  under  the  control  of  the  developers.  .Any  change  can  be 
made  at  any  point  in  the  lifecycle,  provided  that  the  requirements  are  still  satisfied 
and  that  the  change  is  propagated  through  the  design  for  consistency.  The  purpose 
of  specification  and  design  is  to  arrive  at  a  program  that  demonstrably  satisfies  the 
stated  requirements.  In  other  words,  from  a  V&cV  perspective,  the  system  specifi¬ 
cation  and  design  .serve  primarily  to  replace  a  large  verification  problem — directly 
verifying  that  a  systetn  that  executes  the  program  will  satisfy  the  requirements — by  a 
series  of  smaller  verification  problems — verifying  that  the  system  specified  will  satisfy 


the  requirements,  verifying  that  each  refinement  ol  the  design  preserves  satisfaction  of 
tlie  specification,  and  finally  \erifying  that  the  cotie  correctly  implements  the  detailed 
design.  Note  that  this  is  quite  consistent  with  the  more  standard  system  develop¬ 
ment  view  that  successive  reiiuemeiit  of  the  specification  makes  it  ea.sier  to  produce 
code  that  satisfies  the  re<|uir<'ments.  Stepwise  refinement  makes  development  easier 
precisely  because  attention  can  be  focussed  on  the  limited  number  of  factors  that  ar<' 
relevant  to  verification  of  each  small  step. 

.\s  the  systent  changes  over  time,  and  the  specification  and  design  evolve,  lifu  ing 
replaced  the  large  verification  gap  by  many  small  verification  gaps  greatly  simplifies 
irrenfication  of  the  system,  rypically,  a  small  change  to  the  code  requires  only  that 
a  few  of  the  design  refinement  steps  be  modified  and  reverified.  If,  on  the  other  hand, 
the  code  was  directly  verified  against  the  requirements,  a  small  change  to  the  ( ode 
can  have  a  large  impact  on  the  proof,  making  reverification  nearly  as  e\'[)ensive  as 
the  initial  verification. 

However.  a.s  has  Ijeen  noted  several  times  al)ove.  stating  the  requirements  for  the 
typical  .AI  system  is  hard.  .As  a  result,  requirements  definitions  tend  to  be  incomplete 
and  somewhat  vague,  which  makes  a  <  onvincing  verification  of  a  system  specification 
impossible.  In  cases  where  there  is  such  a  gap,  it  may  be  tempting  to  ignore  system 
verification,  and  to  concentrate  on  validation.  It  might  be  argued  that  in  any  ca.se 
where  testing  can  provide  sufficient  evidence  that  requirements  are  satisfied,  there  is 
no  need  for  a  specification  or  design  beyond  the  source  code  plus  comments.  How 
often  have  you  been  told  "The  code  is  self  explanatory?”  The  main  arguments  against 
this  view  are  purely  practical. 

First,  writing  code  involves  making  design  decisions,  whether  those  decisions  are 
recorded  or  not.  The  code  says  how  things  are  done,  but  not  why  they  are  done 
that  way.  .As  a  result,  when  it  comes  time  to  modify  the  code,  it  is  far  from  clear 
whether  a  particular  design  decision — say,  the  choice  of  a  particular  data  structure  or 
algorithm — was  dictated  by  the  necessity  of  satisfying  some  requirement,  or  whether 
it  was  more  or  less  arbitrarily  chosen  from  among  a  number  of  alternatives.  .Suppose 
that  a  stringent  performance  requirement  could  be  satisfied  by  replacing  one  sorting 
routine  by  another.  Further  suppose  that  the  original  sort  weis  stable,  but  that  the 
proposed  replacement  is  not.  Can  the  latter  be  freely  substituted  for  the  former,  or  is 
-.tability  a  necessary  property  of  the  algorithm?  If  the  sorting  unit  was  specified,  the 
question  can  be  answered  by  a  quick  examination  i>f  the  design  document;  if  not.  the 
best  you  can  do  is  make  the  substitution,  run  a  few  tests,  and  hope.  Thus,  virtually 
any  change  to  the  system  requires  extensive  revalidation. 

Second,  unless  there  is  a  system  design  process  that  guides  development,  the  system  is 
likely  to  be  ill-structured,  consisting  of  patches  on  patches,  a  result  of  a  series  of  what 
were  perceived  to  be  the  minimal  changes  necessary  to  eradicate  bugs.  Without  a 
design  breakdown,  any  subsequent  change  to  the  system  requires  complete  retesting. 
There  is  no  way  to  guarantee  that  significant  effects  of  the  change  are  confined  to  a 
particular  program  unit.  But  a  \alidated  design  gives  the  developer  the  capability  to 
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swap  one  unit  for  another,  proviclerl  that  it  satisfies  the  same  specification.  Thus,  a 
solid  sv'^'em  specification  and  design  simplify  the  revali<lation  process.  .AI  systems 
(end  l<.  ..  es|)ecially  comple.x.  especially  liable  to  modification,  and  especially  dil- 
licult  to  verify  and  validate.  Since  specification  and  design  documents  are  essential 
to  reducing  the  cost  of  reverification  and  revalidation,  it  is  especially  important  to 
develop  and  maintain  specifications  and  designs  ol  .A1  systems  . 

Guideline  6.5  In  ■•specification  and  design,  focus  on  reducing  the  cost  oj  rernlidat ion 
and  r(  re  rificnl ion. 


It  must  be  pointed  out  that  only  in  exceptional  cases  can  validation  testing  re|)lace 
verification,  anyway.  .As  Dijkstra.  among  others,  has  rep<'at<>dly  emphasized,  testing 
<  an  only  show  that  bugs  are  present,  not  that  they  are  al)sent.  F.ven  if  confidence  in 
the  correctness  of  the  specification  is  lacking,  validation  can  still  increase  coniiilence 
that  the  system  is  robirst.  reliable,  and  comparatively  free  ot  the  ivpical  ■■meclianical" 
errors — >uch  as  being  olf-by-one  in  some  indexing — made  by  programmers  who  flo  not 
have  a  detailed  design  to  work  from. 

.Assuming  that  the  desirability  of  specifying  tlie  system  and  its  parts  has  been  estab¬ 
lished.  the  next  questions  that  must  be  addressed  are:  What  specification  method 
should  be  used?  How  much  formality  is  desirable?  What  is  the  added  value  provided 
by  so-called  "executable  specifications?’’ 

The  ad\antages  of  formal  specifications  of  system  functionality  over  informal  specifi¬ 
cations  are  substantial.  Recall  that,  from  the  V&V  perspective,  specifications  are  in¬ 
troduced  to  support  validation.  Informal  specifications,  whether  presented  in  natural 
language  or  using  suggestive  diagrams  that  cannot  be  assigned  any  j)recise  meaning, 
can  l)e  very  useful  in  attempting  to  understand  the  system.  But  informal  specifica¬ 
tions  only  support  informal  verification  arguments,  which  are  typically  lengthier  and 
less  convincing.  The  objective  is  to  guarantee  that  any  system  that  meets  the  refined 
-pecificalion  will  meet  the  original  specification,  and  imprecision  in  the  original  or 
refined  specification  makes  this  exceedingly  difficult.  Moreover,  valiflation  of  refine¬ 
ments  of  formal  specifications  can  be  at  least  partly  automated.  .Most  tools  based  on 
|)opular  diagrammatic  methods,  such  as  Structured  .Analysis  [dlj.  perform  at  least 
-ome  consistency  checking  across  levels.  Tools  that  support  textual  specification  lan¬ 
guages  provide  even  greater  support,  based  on  theorem  prov  ing  capabilities. 

Diagrammatic  methods  are  definitely  more  popular  than  textual  methods,  principally 
because  the  average  system  designer  finds  them  easier  to  use  and  the  average  system 
implementor  finds  them  ejisier  to  understand.  However,  full  specifications  of  the 
behavior  of  complicated  systems  in  diagrams  are  completely  incomprehensible  and 
unmaintainable,  often  amounting  to  an  attempt  to  represent  every  state  of  the  system 
by  a  box  and  every  state  transition  by  an  arrow.  .As  a  result,  specificatioit  diagrams 
fend  to  require  abstraction  that  suppresses  detail.  This  suppression  of  detail  limits 
the  scope  of  verification.  .Since  developers  of  . A  f  systems  are  comfortable  with  formal 
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text  ual  representations  of  symbolic  information  and  techniques  for  manipulating  such 
representations — that  is  what  AI  programs  do! — text-based  specilicatioiis  (or  liyljiid 
methods  that  combine  diagrams  with  formal  annotations)  would  seem  superior  for 
A I  d<‘velopment. 

Guideline  6.6  For  A  I.  te  rt-hased  ■•^pecificattons  are  supfrior  to  linKji  nni-hn^f  d  ^pen- 
ji cal  Kills. 

lextiial  formal  specification  languages  can  be  divided  into  two  types.  First,  tliere  are 
iilgfhrnic  specification  languages,  such  as  OB.J  [IS]  and  .A('’T  [IT],  In  these  languages, 
one  specifies  a  collection  of  modules,  each  of  which  consists  of  ob  jects,  opei  ations  on 
those  objects,  and  equations  those  operations  satisfy.  Second,  there  are  niodfl-lmsfil 
languages,  such  as  VDM  [o]  and  Z  [40|.  In  these  languages,  one  specifies  modules  that 
are  bifilt  up  from  mathematical  components — sets,  .sequences,  ndatioiis.  fuin  tloiis. 
.uul  Si)  on — using  a  logic-based  latiguage.  The  essential  difference  betweim  the  two  i- 
that  the  former  provides  a  partial  description  of  the  unit  while  the  latter  i)ro\  ides  a 
mathematical  model  of  the  unit. 

.Mthough  there  are  theoretical  reasons  for  preferring  one  type  i)f  formali;'ation  over 
the  other,  in  practice  there  seem  to  be  few  grounds  for  choositig  betwi'en  them.  Some 
specificattons  have  been  written  up  both  algebraically  and  set  theoretically — e.g..  a 
partial  specification  of  the  (.’nix  file  system  [4,  22] — and  it  seems  that  translating 
between  the  two  approaches  is  straightforward.  The  set  theoretic  approach  is  more 
widely  used  in  industry,  and  the  support  tools  available  for  the  set  theoretic  ap- 
l)roach  are  definitely  more  mature.  On  the  other  hand,  the  algebraic  approach  tends 
to  restrict  itself  to  logical  languages  for  which  efficient  mechanical  tln'orem  proving 
is  possible,  and  thus  offers  greater  long  term  potential  for  automaterl  verification 
support.  If  you  are  unfamiliar  with  the  details  of  the  two  api)roaches.  examine  a 
good  introductory  survey  (15.  43]  and  make  your  own  decision.  But  be  aware  that 
the  state-of-the-art  is  evolving  rapidly,  so  the  particular  systems  described  in  these 
surveys  may  no  longer  be  the  best  candidates  by  the  time  you  read  this  report. 


Guideline  6.7  The  state  of  the  art  in  algebraic  and  model-hn'^nl  spi  np'i  ation  is  i  rolr- 
nig  rapidly.  Make  sure  you  have  considered  the  best  current  randuinte  tools  htjon 
rhoosing  between  the  two  approaches. 

Some  programming  languages  claim  to  be  executable  specification  languages.  The 
idea  is  that  they  provide  the  capability  to  interpret  or  compile  a  formal  specification 
language,  thus  combining  the  advantages  of  a  specification  and  a  prototype.  It  seems 
clear  enough  that  writing  a  specification/prototype  is  likely  to  be  less  work  than  writ¬ 
ing  a  specification,  writing  a  prototype,  and  verifying  that  the  prototype  matches  the 
relevant  part  of  the  specification.  REFINE  [28],  for  e.xample.  provides  much  of  the 
functionality  of  Z.  but  transforms  its  specifications  into  LiSP.  In  this  case,  the  cost  of 
executability  is  a  weakened  set  theory — all  of  RefiNE's  sets  are  finite,  but  Z  sup|)ot  ts 


description  of  infinite  sets — and  some  deviation  from  logical  purity  for  the  sak('  of 
efficiency.  .Also,  use  of  an  e.xecutable  specification  language  tends  to  encourage  Dver- 
specification.  as  a  less  abstract  spec  ification  will  e.xecnte  more  efficiently,  pioviditig  a 
more  useful  prototype.  Thus  design  derisions  can  creep  into  tlie  specification,  lesidt- 
ing  in  a  commitment  to  design  features  tliat  are  appropriaie  for  the  prototype  but 
inappropriate  for  the  deliverable  system.  Again,  the  best  approach  is  to  explore  the 
capabilities  of  the  systems  available  at  the  time  you  read  this,  and  decide  lor  yourself 
whether  any  is  well-suited  to  your  needs. 

Guideline  6.8  f^.s/  UQ  an  fxeculahle  <peciJicntion  language  entails  the  risk  of  orer- 
specification.  but  offers  substantial  benefits.  Whether  the  benefits  balniKt  Hu  nsk.'^ 
depends  on  the  project  and.  the  choice  of  e.recntnble  specification  language. 


7.  For  Designers:  Designing  Your  Software  to  Support  V'&:V 


7.1  Introduction 

riie  principal  arlvice  we  have  tor  system  designers  is 

Guideline  7.1  To  jncdihitf  17-1'.  krep  Iht  design  ns  simple  ns  possible. 

In  this  section,  the  meaning  of  "simplicity'’  is  spelled  out  in  great  detail,  and  i  he 
possibility  of  tiefining  tpiantitative  mea.snres  of  design  simplicity  is  explored.  .\lso. 
the  relative  advantages  of  formal  and  informal  design  techniques  are  I'onsideital. 

.Vote  that,  since  design  iiuolvt's  specification  of  parts,  the  material  on  s|>c(  iheat  ion 
in  Section  6.2  is  etiuallv  relevant  to  design. 


7.2  What  is  Simplicity? 

The  notion  of  design  simplicity  seems,  at  first  glance,  to  tlepend  on  the  design  method 
that  is  being  employed.  Consider,  to  begin  with,  classical  design  techniques  based  on 
functional  decomposition.  .A  good  heuristic  in  using  these  function-oriented  ntethods 
is:  Minimizt  the  namher  oj  nilerconnections  of  subfunctions.  In  other  words,  the 
design  represented  by  the  boxes  and  arrows — boxes  are  subfunctions  and  arrows  rep¬ 
resent  data  flow — in  Fig.  7-1  should  he  preferred,  all  else  being  e(|ual.  to  the  design 
in 

Fig.  7-2. 

The  criteria  for  judging  the  cpiality  of  an  object-oriented  software  design  seem  to  be 
quite  different.  The  principal  heuristic  in  object-oriented  design  is:  Let  the  mt.--.snije.^ 
to  objects  sag  "ichnt”  ts  to  he  done:  ns.socinte  "how”  to  do  it  with  the  most  geiirnil 
rinsses  possible.  .Judged  i)y  the  criterion  for  classical  design,  good  object-oripnt('d 
designs  will  fare  rather  poorly,  as  they  tend  to  be  highly  interconnected. 

Observe,  however,  that  reducing  the  iuiml)er  of  connections  among  subfunctions  tends 
to  minimize  information  flow  among  subfunctions.  Indeed,  reduction  of  information 
flow  seems  more  fundamental  than  reducing  the  number  of  information  channels,  sini  e 
reducing  the  number  of  channels  simply  by  passing  more  complex  data  structures  does 
not  improve  the  design.  .And  also  note  that  the  object-oriented  design  heuristic  tends 
to  reduce  information  How  across  abstraction  boundaries:  many  messages  flow  around 
the  system.  i)ut  each  contains  little  information.  Based  on  these  considerations,  the 
following  seems  to  be  a  ijenernihj  appropriate  design  heuristic. 

Guideline  7.2  .Vo  nintter  what  design  method  you  mny  be  using,  minimize  informa¬ 
tion  flow  nrio.s.s  module  absl rnction  honndnries. 


Figure  7-1:  A  Possible  Functional  Decomposition 


Figure  7-2:  An  Alternative  Functional  Decomposition 


After  all.  one  of  the  signs  that  you  have  i<lent,i(ie<l  a  natural  boundary  is  that  the 
whole  can  be  cleft  relatively  cleanly  into  parts  at  that  boundary. 

The  consecjuences  of  using  inforniation  (low  as  a.  lueasiire  ot  simplicity  will  now  be 
<  (Xi.sidered. 


7.3  Measuring  Simplicity 


Initially,  consider  the  case  of  a  knowledge-based  system  that  calculates  probabilities 
of  hypotheses  from  evidence  using  a  network  of  inference  meclianism.s  that  update 
l>robability  distributions  over  their  outputs  base<l  on  changes  to  the  probability  dis- 
iribution  over  their  inputs.  .A  typical  system  of  tliis  sort  is  shown  in 

Fig.  7-1. 

Shortly  after  Shannon  inaugurated  the  study  of  Information  riieory.  Carnap  anti 
Bar-Hillel  [ll]  showed  that  a  formal  analogue  of  Shannon  s  information  measure 

I(</)  =  “H  logif'/j) 

J 

provides  a  natural  e.xplication  of  amount  of  semantic  in  formal  ion  when  the  probabil¬ 
ities  Hj  are  interpreted  as  justified  degrees  of  belief.  Thus  the  amount  of  information 
associated  with  an  inference  mechanism  that  transforms  prior  prol.ability  distribution 
<!  on  its  output  into  posterior  probability  distribution  r/  is  simply  -  K^j.  .\nd 
-^o.  given  a  design-level  probabilistic  model  of  the  systems  inferential  behavior,  the 
('.\pectecl  intermodule  information  flow  can  be  computed. 

The  fletails  of  how  to  calculate  this  measure  depend  on  the  |)articular  inference  mech¬ 
anism  used.  One  might  hope  that  in  cases  where  the  update  algorithm  is  reasonably 
clficient  (e.g..  the  recently  developed  network-based  algorithms  for  Bayesian  infer¬ 
ence  [.'12].  or  even  more  general  algorithms  for  cross  entropy  minimization  [3Sl )  a 
reasonably  efficient  analytic  determination  of  expected  information  flow  would  be  pos¬ 
sible.  .Note,  however,  that  this  calculation  re<iuires  information  not  usually  available 
prior  to  implementation;  having  all  the  knowledge  recpiirevl  to  actually  perform  infer¬ 
ence  when  the  design  is  completed  is  the  exception  rather  than  the  rule  in  knowledge- 
based  system  development.  Fortunately,  expected  information  flow  can  always  be 
("^timated  by  simulation  in  a  completely  straightforward  fashion  at  design-time  us¬ 
ing  the  high-level  probabilistic  model  based  on  estimates  of  the  average  information 
added  by  each  inference  mechanism. 

The  next  question  that  must  be  addres.sed  is:  How  can  Ihift  he  generalized  to  other 
.>orl.s  of  .■^gstemsf  If  quasi-probabilistic  mechanisms  are  used,  the  generalization  is 
(^asy.  But  the  generalization  to  systems  that  perform,  say.  deductive  inference  is 
not  so  ea.sy.  The  problem  is  that,  according  to  the  standaifl  definition  of  semantic 
information,  deduction  does  not  yield  new  information:  if  you  assign  A  a  probability 
of  1  and  .1  implies  B.  then  8  must  also  be  a.s.signed  a  probability  of  I  to  maintain 
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Figure  7-.]:  An  Inference  Mechanism 


Figure  7-4:  A  Network  of  Inference  Mechanisms 


consistency.  So  what  is  needed  is  a  sense  of  information  such  that  if  yon  initially 
know  tliat  A  but  not  that  B  and  you  later  succeed  in  deducing  D  from  A.  von  have 
gained  information.  ' 

rims,  defining  a  (|uantitativ(' measure  of  sim()licity  that  can  provide  design-ti nu  guid¬ 
ance  in  the  (piest  to  support  V&V  is  only  a  partially  solved  i)roblem.  Ibit  in.  r  y  AI 
systems  are  of  the  probabilistic  sort  where  a  useful  measure  has  been  already  Ix'en 
ch'fined . 


Guideline  7.3  If  a  simplir/hj  measurf!  is  available,  use  it  to  emhuitf  Hu  n-tuhn 
■■cniliHi  ity  of  rniHcnlh/  diljf  i-nit  design.^. 


7.4  Formal  vs  Informal  Designs 

If  the  sole  purpose  of  documenting  designs  were  to  provitle  guidance  tc;  the  iin[)le- 
mentors.  then  the  defects  of  using  an  informal  or  semi-formal  design  method— natural 
language.  Structured  .Analysis.  Structured  Design,  and  Commercial  Products  such  as 
S.\DT.  IDEF.  etc. — may  be  balanced  by  the  main  advantage;  any  programmer  can 
understand  designs  presented  in  these  formalisms  with  little  or  no  training.  Hut  when 
strong  verification  of  the  code  against  the  design  is  required,  the  advantages  of  a  more 
formal  design  dominate.  In  fact,  since  design  is  decomposing  the  system  into  parts, 
specifying  those  parts,  decomposing  those  parts  into  subparts,  and  so  on.  the  advan¬ 
tages  of  formal  design  are  e.xactly  the  same  as  those  of  formal  specification,  already 
described  in  Section  6.2. 


'  \\V  explored  three  different  approaches  to  explicating  tins  more  generally  applical)le  roncepi 
of  information. 

•  Taking  a  semantic  approach,  we  employed  a  Ijroader  class  of  models  ( inchuling.  ^ay. 
urn  models)  that  distinguish  between  non-trivially  logically  equivalent  .sentences,  ,and 
left,  the  definition  of  information  remains  the  same. 

•  Taking  a  syntactic  approach,  we  measured  the  complexity  of  the  simplest  derivation 
of  B  from  .4  and  used  that  as  the  measure  of  information  gained  in  inferring  B  from 
.1. 

•  Taking  an  epistemic  approach,  we  used  a  "fine  grained"  epistemic  logic  and  di'tined 
a  new  concept  of  information  in  terms  of  the  old 

r{.4)  =  I(Knows( system,  .4)) 

VVe  then  showed  that  the  semantic  and  syntactic  approaches  are  special  cases  of  the  epistemic 
approach,  so  our  subsequent  research  focus,sed  on  the  epistemic  definition  of  information 
(This  one  of  a  number  of  cases  where  mainstream  AI  techniques  may  have  enormous  impact 
on  the  VAV'  problems  raised  by  AI.)  .At  the  tune  of  the  writing  of  this  manual,  the  research 
is  not  sufficiently  mature  to  provide  any  immediate  guidance  on  how  to  measure  information 
flow  in  non-probabilistic  systems,  but  it  does  show  considerable  potential. 


8.  For  Programmers:  Choosing  Programming  Techniques 


In  this  section,  we  discuss  the  impact  of  the  choice  of  the  AI  programming  techiii(|ues 
of  Section  3  on  the  appropriateness  and  elfectivcness  of  various  VSc\'  techni(|ues. 
Guidelines  for  choosing  programming  techniques  that  simplify  V'&:V  are  presented. 
Also,  for  each  technique,  we  comment  on  the  difficulty  of  providing  a  given  level  of 
verification,  which  verification  technic|ues  might  usefully  he  employed,  and  how  and 
what  to  test  to  provide  convincing  validation  of  the  system. 

Although  this  list  of  techniques  is  far  from  e.^haustive,  the  main  objective  is  to  provide 
you  with  some  examples  of  how  the  conventional  wisdom  of  V'&V  can  l)e  applied  to 
AI  development,  even  though  many  AI  programming  techniques  are  never  considered 
in  \  &z\  textbooks.  If  a  programming  technique  of  interest  is  not  listed,  you  should 
be  able  to  adii  it  by  using  the  following  techniques  as  models. 


8.1  Data-Driven  Programming 

Recall  that  the  essence  of  data-driven  programming  is  that  data  triggers  the  retrieval 
and  running  of  associated  programs.  Typically,  these  techniques  will  be  used  in  a 
situation  where  a  specification  specifically  calls  for  a  certain  function  to  be  performed 
for  a  certain  independently-specified  program  to  be  executed  when  the  data  satis¬ 
fies  a  certain  condition.  Thus,  the  "incremental”  verification  problem  introduced  by 
the  use  of  data-driven  programming  is  showing  that  the  data-program  association  is 
correct  according  to  the  specification,  and  that  the  condition  on  the  data  is  always 
tested  when  appropriate.  (This  observation  explains  why  data-driven  programming 
is  such  a  useful  technique.  .Saying  something  like  "  WTicnerer  the  data  satisfy  condi¬ 
tion  C .  then  do  A’’  is  a  very  natural  way  of  specifying  desired  behavior.  Verification 
of  the  code  that  tests  the  condition  on  the  data  is  generally  easy,  as  that  code  is 
simply  an  optimization  of  a  nai’ve  implementation  of  the  condition,  and  the  verifica¬ 
tion  of  the  associated  programs  must  be  performed  anyway.  So  use  of  uata-driven 
programming — as  opposed  to  more  traditional  techniques  that  do  not  treat  the  condi¬ 
tions  and  programs  as  "first-class  objects” — is  straightforward,  because  the  structure 
of  the  code  more  closely  matches  the  structure  of  the  specification.) 

So  our  first  guideline  on  choice  of  programming  techniciues  to  support  the  \  .V:\ 
process  is: 

Guideline  8.1  L'se  data-driven  proyrnmining  when,  but  only  ivlien.  the  form  of  the 
■specification  naturally  calls  for  it. 


If  this  guideline  is  adhered  to.  the  incremental  validation  problem  can  usually  be  made 
tractable,  even  if  very  strong  foiimal  validation  is  required.  Validation  of  the  data- 
program  association  should  be  trivial,  whether  the  association  is  explicitly  represented 


I!) 


in  a  data  structure,  implicitly  represented  by  inclusion  of  a  method  definition  wit  hin 
the  lexical  scope  of  a  class  declaration,  or  whatever.  No  matter  what  programming 
language  mechanism  is  used  to  create  the  a,ssociation.  exactly  whal  data  is  as.sociated 
with  what  program  should  be  clear  enough.  The  potentially  difficult  part  ol  the 
validation  is  assuring  that  the  tests  are  always  applied  when  appropriate.  Probabl} 
the  most  straightforward  solution  is  to  use  some  sort  of  active  data  stru<  tures  with 
a  well-defined  interface,  so  that  the  code  for  testing  the  condition  on  the  data  can 
l)e  integrated  with  the  code  for  changing  the  data.  Most  modern  languages  (Ll.sp. 
C++.  .\da.  and  so  on)  provide  support  for  encapsulating  data  structures,  so  this 
straightforward  solution  can  be  easily  implemented. 

Guideline  8.2  Encapsulate  data  structures  that  contain  data  until  associnted  prin  t - 
dares,  using  the  appropriate  linguistic  ineclinnistn  (ij  there  /.s  one). 


One  prominent  case  when  these  techniques  cannot  be  applied  is  when  eitfier  (  1 )  Plto- 
LOG  is  used  and  the  data  associated  with  programs  is  simply  a  subset  of  t  he  PROLOG 
database,  or  (2)  a  shell  for  building  rule-based  systems  is  used,  the  data  associated 
with  programs  is  part  of  the  database  the  rules  operate  on.  and  the  database  does 
not  support  attachment  of  daemons  to  the  data.  (For  present  purposes,  a  dm  mon 
is  simply  a  procedure  that  can  be  associated  with  data  that  will  e.xecute  when  the 
data  is  changed.  That  is,  it  is  a  b<isic  data  management  mechanism  for  supporting 
data-driven  programming.)  In  this  case,  the  best  solution  is  to  rewrite. the  program 
so  that  the  data  is  not  stored  in  the  PROLOG  database.  If  this  is  infeasil)le — because, 
for  example,  the  data  is  being  used  in  computational  reflection — the  next  simplest 
solution  is  to  distribute  the  data-program  association  among  the  individual  PROLOG 
program  clauses  or  production  rules.  For  example,  if  the  clause 

might  i  liange  the  Prolog  database  so  as  to  make  condition  C  true,  and  if  A  is  to  i^e 
performed  whenever  C  becomes  true,  the  clause  should  be  rewritten 

[A  .v.g]  t7i(.c,c,),  Q2(-i\ -2) .  ■  ■  ■  .  Q,.(.r.c„),  (  C  ->  .1  )  . 

Tlie  incremental  verification  problem  then  becomes  assuring  that  the  conditionals 
have  been  added  everywhere  they  are  required — which  can  require  an  arbitrarily  com¬ 
plex  analysis  of  which  clauses  might  make  conditions  on  the  data  true  when  they  are 
called.  This  may  not  be  difficult  in  a  particular  case,  but  the  following  guideline 
provides  sound  general  advice. 

‘Till*  ii.sp  of  for  1/  and  ,  for  miti  is  standard  Edinbiirgli  Prolog  notation.  \\V  will  also 
use  ->  for  the  niore-or-less  standard  conditional  operator. 


■)0 


Guideline  8.3  Avoid  using  data-driven  prog  ramming  when  the  data  is  storrd  m  a 
glohnlly  accessible  database  that  does  not  support  attnrh mrnt  of  daemons  to  ttie  dnta. 


All  alternative,  in  the  case  of  PROLOG,  is  to  hnilcl  clata-driven  |)rograinmiii^  su|)()oi  t 
into  a  ineta-inter[)ieter.  This  is  an  elegant  solution,  hut  typically  far  harch'i  to  vinify 
than  the  rewritten  rules. 

I  se  of  (.lata-driven  programming  does  not.  much  alfect  system  validation.  WIhmi  a 
data-prograin  a.ssociation  is  dictated  by  the  ret|uirements.  the  correrlnes.s  of  tlu'data- 
program  association  will  be  ade(|uately  demonstrated  in  e.Ktended  use.  The  priiu  ipal 
adtled  burden  in  validation  is  making  sure  that  there  are  no  cases  when,  given  the  dal  a. 
some  program  should  have  been  called,  but  was  not.  If  the  data  can  be  moniit)ied 
independently — by  running  under  a  debugger,  for  example- — sampling  \alues  may 
raise  the  level  of  confidence  in  the  completeness  of  the  implementation  of  the  daia 
program  association. 

8.2  Discrimination  Nets 

Discrimination  nets  are.  in  effect,  an  optimization  of  a  specified  inore-or-less  "llat" 
categorization.  That  is,  the  specification  of  the  categories  will  probably  look  some¬ 
thing  like 

//  P\{  v).  then  x  is  a  Ci.  If  Pi{x).  then  r  is  a  C’>.  //  PAx).  then  x  is  n 
C n  ij  Qsf-f)'  ffi'e  X  IS  a  C  m-  ... 

.\  rather  tormal  verification  that  the  categorization  implicitly  defined  by  the  ti'sts 
in  the  net  correctly  correspond  to  the  specified  categories  is  usually  feasible.  The 
optimization  used  to  build  the  tree  amounts  to  observing  that  there  are  logical  cle- 
peiulencies  among  the  properties,  so  the  results  of  previous  tests  can  be  used  to 
simplify  subsequent  tests.  What  must  be  verified  is  that,  if  category  C  is  associated 
with  leaf  node  A’,  then  the  series  ro(.r)  =  tq,  Ti(x)  =  /q.  Tiix)  =  r2, ...  of  test  results 
associated  with  any  path  leading  to  .V  implies  that  x  is  indeed  C.  Since  the  re<;rs 
were  chosen  so  as  to  have  this  property,  it  should  be  easy  enough  to  show  that  tliev 
do. 

Guideline  8.4  When  optimizing  the  tests  in  the  net.  consider  not  only  run-time 
efficiency,  but  the  difficulty  of  demonstrating  correctness. 


One  of  the  advantages  of  this  programming  technique  is  that  a  relatively  limited  set 
of  test  data  can  provide  a  great  deal  of  confidence  that  the  classification  is  correct. 
Generally,  the  level  of  confidence  rises  more  quickly  if  the  intermediate  categories 
defined  by  partial  test  results  are  natural,  rather  than  invented.  More  intportnnt. 


fevalidation  is  simplified  alter  making  the  most  common  <  liaage  to  the  net .  adding  new 
(.  ategories  (including  refining  existing  categories),  riiis  can  l)e  achieved  by  <is.sociating 
each  category  C  with  a  collection  { F(,.  Fi.  F,, . . .}  of  independent  primitive  Features 
that  jointly  guarantee  membership  in  C.  and  then  treating  the  optimization  problem 
as  being  one  of  ordering  the  rests  for  the  F,.  In  the  ideal  ca.se.  each  initial  segment 
of  the  ordered  list  of  features  will  determine  a  natural  category,  i.e.,  each  test  will 
determine  which  natural  subentegory  of  a  natural  category  the  given  information 
belongs  to.  (This  approach  will  pro<luce  discrimination  trees,  rather  than  sometimes 
more  elficient  general  nets.)  In  other  words,  the  discrimination  is  based  on  a  natural 
cla.ssification  hierarchy. 

Guideline  8.5  When  i>o>sible.  hn.st  the  discnmuint inn  on  a  natural  rlns.-nlit  <il ton 
h  If  rarch  I). 

8.3  Meta-Level  Control  Structures 

The  principal  verification  burden  imposed  by  meta-level  control  mechanisms  is  that 
Common  Lisp  programs  containing  undisciplined  uses  of  function  can  l)e  dilficitlt 
to  reason  about  itsing  conventional  specification  formalisms.  Typically,  specifications 
will  either  include  a  very  procedural  representation  of  the  algorithm  to  be  used,  or  will 
be  formalized  more  abstractly  in  a  higher-order  or  set-theoretic  specification  language. 
Therefore,  the  problem  cati  be  minimized  by  using  only  closures  that  need  not  close 
over  any  variable  bindings,  itt  which  case  the  closures  can  be  treated  as  mathematical 
functions  that  contain  no  information  about  the  environment.  The  same  can  l)e  said, 
mutatis  mutandis,  about  the  u.se  of  Scheme's  procedures. 

Guideline  8.6  When  iinplf  mcntiny  ine.fa-level  control  -Structures,  avoid  asimj  cto^mij 
over  functions  that  contain  free  variables,  to  simplify  verification. 

.\  major  advantage  of  using  meta-level  control  structures,  from  the  standpoint  of 
\k\'.  is  that  some  requirements  on  the  functioning  of  the  system  might  only  be 
verifiable  via  proving — either  formally  or  informally — that  its  control  structure  ha.s 
certain  desirable  properties.  For  example,  we  might  require  that  a  system  which 
contains  several  routines  that  might  be  applied  in  problem  solving  to  select  among 
them  fairly.  Proving  such  properties  is  often  much  easier  when  control  >tructur<‘s  are 
explicitly  represented  at  the  meta-level. 

Guideline  8.7  If  verification  requires  proving  that  the  program  's  control  structurf 
has  certain  properties,  consider  representing  control  explicitly  using  meta-level  control 
structures  to  simplify  verification. 

Validation  of  systems  that  make  use  of  meta-level  control  can  become  more  com¬ 
plex  because  there  is  no  longer  a  meaningful  distinction  between  data  and  control. 
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A  formal  requirement  for,  say,  exhaustive  branch  testing  cannot  lie  satisfied,  be¬ 
cause  new  branches  are  created  at  run-time  l>ased  on  the  data.  From  an  unsym¬ 
pathetic  v  iewpoint,  standard  implementations  of  meta-level  control  structures  miglil 
be  compared  to  self-modifying  code-  which  is  thought  to  l)e  difficult-to-impossil>le 
to  strongly  valivlate — in  this  respect.  Rather  than  relying  on  the  usual  criteria  for 
determining  the  extent  of  validation,  ad  hoc  arguments  that  the  desired  level  of  vali¬ 
dation  has  been  achieved  will  be  necessary.  More  effort  must  be  put  into  defining  ami 
defending  validation  procedures  when  meta-level  control  mechanisms  are  employefl. 
(Those  (Jiocedures  may  not  be  ('specially  ilifficull  or  expensive  to  perform,  however.) 

Guideline  8.8  Although  stnudavd  criteria  for  comprehensiue  validation  mug  be  huid 
to  ■'lutisfg.  nn  pie  me  at  at  ion. s  with  meta-level  control  .A  ructure.^  are  not  nect.^e^anlg  ihj- 
ficult  to  cnlidate  (i.e.  it  Jriquentig  cs  not  difficult  to  e.terci.'ee.  .<uch  .>tructurr.<i  in  ivnij.-^ 
that  proride  con.^uie rnble  ronjide  nee  that  theg  perform  fl.s  theg  .should).  . 


8.4  Deductive  Information  Retrieval 

Deductive  information  retrieval,  being  based  on  formal  logical  deduction,  can  be  an¬ 
other  good  candidate  for  formal  verification.  But,  as  was  noted  in  the  description  of 
the  technique,  such  systems  typically  deviate  from  "logical  purity"  in  some  way.  and 
the  extent  and  nature  of  the  deviation  can  influence  the  efficacy  of  formal  verification 
techniques.  First,  there  may  or  may  not  be  a  mathematical  semantics  that  determines 
whether  derivations  are  correct  or  not.  .Second,  if  there  is  a  formal  semantics,  the 
deductive  technique  may  or  may  not  be  complete  with  respect  to  that  semantics,  i.e.. 
the  deductive  technique  might  not  be  strong  enough  to  derive  all  conclusions  that  fol¬ 
low  from  the  facts  stored  in  the  knowledge  base.  Third,  the  deductive  technique  may 
or  may  not  be  sound  with  respect  its  formal  semantics,  i.e..  the  deductive  technique 
might  allow  incorrect  conclusions  to  be  drawn  from  the  knowledge  base. 

Guideline  8.9  If  the  extent  of  the  conse<iuence  relation  /.s  under  gour  control,  both 
renfication  and  validation  will  be  grentlg  simplified  if  gou  rhoo.se  a  relation  based  on  <i 
mathematical  .semantics,  rather  than  one  that  ran  onlg  be  defined  in  procedural  term.-. 

Guideline  8.10  .Make  the  deduction  trchnique  as  close  to  complete  ns  perjormn nee 
constraints  allow.  .And  make  sure  that  all  obvious  conclusions  will  be  drawn  from 
the  knowledge  base,  because  missing  obvious  conclusions  will  make  the  user  doubt  the 
utilitg  of  the  sg.stem. 

Guideline  8.11  Be  extremelg  hesitant  to  give  up  on  soundness.  Even  if  gou  ran 
argue  that  no  incorrect  conclusions  will  be  drawn  in  practice,  use  of  an  unsound  de¬ 
duction  technique  will  tend  to  undermine  the  user's  confidence  in  the  sgstem,  making 
ralidntton  and  re  validation  much  more  expen.-ive  and  difficult. 


Whether  a  “nice”  deductive  technique  exists  is  often  determined  by  the  specification. 
If  the  specification  determines  exactly  what  conclusions  should  follow  from  the  facts 
in  the  knowledge  base,  the  programmer  must  implement  that  particular  notion  of 
con.ser|uence,  no  matter  how  messy  it  may  l)e.  ( .Specific  at  ions  sometimes  nnderdeter- 
mine  the  consecjuence  relation.  l)ecause  a  good  specifier  knows  that  most  stattdard 
notions  of  logical  consequence  are  too  expensive  to  implem'  ‘  efficiently  and  chooses 
to  leave  the  details  to  the  programmer's  discretion.)  t 'ornpi  '  ing  logical  purity  for 
efficiency's  sake  is  a  practical  necessity,  and  tiie  desired  effects  of  a  compromise  are 
nsTially  impossible  to  express  except  in  form  of  an  algorithm.  So  most  specifications 
will  |)rovide  a  procedural  representation  of  the  desired  consequence  relation,  and  the 
programmer's  task  is  simply  to  implement  that  algorithm.  In  this,  the  most  common, 
case,  there  is  no  special  verification  or  validation  problem  associated  with  the  deduc¬ 
tion  engine;  it  is  sim|)ly  a  matter  of  showing  that  an  algorithm  has  been  implementcfl 
correctly.  Moreover,  sonte  languages  and  shells — most  notably  PnoLOt: -  provide  an 
deduction  engine  wliich  need  not  be  verified  and  is  already  well  validaierl. 

Guideline  8.12  Whether  deductive  information  retrieval  unll  be  used  should  he  de- 
lennnied  bij  the  form  of  the  specification  of  the  system  's  retrieval  capubililif  >. 

There  can  be  a  second  set  of  \X'V  problems  associated  with  deductive  information 
retrieval:  the  general  knowledge  in  the  knowledge  ba-se  must  be  validated.  (Verifi¬ 
cation  is  usually  trivial.  The  general  knowledge  is  given  in  the  specification,  in  a 
form  that  can  be  directly  coded  in  the  knowledge  representation  formalism.)  If.  when 
the  system  is  used,  the  knowledge  base  consists  njainly  of  data,  validation  may  be 
straightforward,  with  just  a  few  carefully  defined  retrievals  being  sufficient  to  show 
that  the  general  knowledge  is  correct.  But  typically,  the  general  knowledge  is  exten¬ 
sive.  consisting  of  hundreds  or  thousands  of  general  facts.  While  there  are  classical 
fechiiiques  for  verification  and  validation  of  such  coliections.  such  as  showing  that  the 
knowledge  base  is  consistent  by  showing  that  it  has  a  model,  and  some  progress  has 
l>een  made  on  defining  analytical  techniques  to  show  that  a  knowledge  base  has  other 
desirable  properties  [41],  there  is  no  substitute  for  extensive  validation  testing.  In 
most  cases,  this  is  not  a  problem  for  the  programmer,  because  the  specification  gives 
the  (possibly  buggy)  general  knowledge,  but  it  is  still  worth  noting  at  this  point. 

Guideline  8.13  The  power  of  deductive  information  rf  triernl  is  that  the  knoirhdije 
rnn  be  combined  in  unexpected  ways  to  draw  •'Urpri.stng  conclusions.  It  is  hnrd  to 
yunrantee  all  these  conclusions  will  be  connect,  even  when  you  can  yiinrantee  that  they 
are  all  consequences  of  the  general  knowledge  and  the  particular  data.  In  other  words, 
it  is  hnrd  to  validate  knowledge  bases. 

8.5  Production  Systems 

Production  systems  represent  a  more  radical  departure  from  conventional  program¬ 
ming  than  the  techniques  discussed  above,  so  much  so  that  our  notion  of  the  "in- 


crementar  verification  and  validation  problems  associated  with  a  prograiuming  tech- 
nitiae  is  not  really  meaningful  in  this  case.  There  is  an  illuminating  analogy  that 
can  l>e  tliawn  with  deductive  information  retrieval,  however.  The  recognize-act  loop 
can  gejierally  be  verified  using  conventional  tethnic|ues.  but  this  code  is  usually  quite' 
simple  t ompared  to  highly  optimized  constructive  theorem  prover.  (This  is  not  in¬ 
variably  true,  liowever;  some  rule-based  system  shells  provide  very  complicate*!  rule 
selection  mechanisms.)  In  addition,  the  recognize-act  loop  is  usually  'given'  by  th*- 
programming  language  or  shell  being  used.  .At  any  rate,  verification  and  validation  of 
the  recognize- a*  t  loop  is  not  the  difficult  problem;  verification  and  validation  of  the 
rules  is. 

The  difficulty  of  verifying  and  validating  a  collection  of  assertions,  fliscus.sed  in  t  he 
previous  section,  is  even  greater  for  rules.  While  a  general  assertion.  If  il  trar  thnl 
-1.  then  it  IS  tnii  that  B.  may  used  as  if  it  were  a  special  case  of  a  prorluctiou  rule.  i.e.. 
as  If  it  IS  trill  that  A.  mid  D  to  the  knowledije  base,  the  verification  teclmicjues  used 
for  rules  do  not  generalize  in  a  straightforward  fashion  to  arbitrary  production  rules. 
.\  rule's  action-part  can  make  an  arbitrary  change  to  some  data  structure.  If  that 
data  structure  in  some  sence  represents  e.xternal  reality,  then  the  change  naturallv 
corresponds  to  an  assertion,  and  the  verification  techniques  used  for  deductive  infor¬ 
mation  retrieval  can  be  employed.  But.  as  experience  with  PROLOG  |)rogramrning 
has  demonstrated,  it  is  not  always  possible  to  find  a  nice  "declarative  interpretation " 
of  transformation  rules. 

Production  rules  can  be  used  to  perform  steps  in  arbitrary  algorithms,  and  it  is 
generally  the  algorithm  or  the  function  computed  by  the  algorithm  that  is  found  in 
the  specification,  not  production  rules.  It  may  be  easy  to  verify  that  a  collection 
of  rules  considered  in  isolation  is  an  implementation  of  some  algorithm  that  can  be 
\erified  against  the  specification,  but  the  verification  that,  when  combinerl  with  other 
rules,  no  problems  will  arise  may  be  completely  infeasible,  .\fter  all.  the  point  of 
the  production  system  architecture  is  to  encourage  surprising  interactions  among  the 
rides. 

Guideline  8.14  If  you  have  a  functional  specification  and  st rang  I  TT'  l  eijuirf  nients, 
production  systems  are  not  a  good  candidate  for  ;niplenientat ion . 

Validation  of  knowledge  bases  is  at  least  somewhat  simplifieil  in  \  irtue  of  useful  proj)- 
crlies  of  deduction.  For  example,  deductive  inference  preserves  truth — if  i  he  prenu.s<'s 
are  true,  the  conclusion  must  also  be  true — and  hence  is  monotomc.  i.e..  new  rlata 
added  to  the  knowledge  base  cannot  undermine  previous  conclusions.  Without  some 
such  properties  associated  with  the  application  of  the  production  rules,  verification 
and  validation  of  a  rule  base  are  practically  impossible. 

Guideline  8.15  Attempt  to  define  invariants  that  capture  nil  re/e  rant  require  iiii  nts 
on  the  function  implemented  by  the  production  system.  If  you  can  dejiiie.  such  inrnn- 
nnts.  the  same  sort  of  techniques  used  in  \  &[  of  deductive  information  retriernl  ran 
be  applied. 


8.6  Frame  Databases 


In  contrail  to  the  iJtlier  piograniming  teclmi(iues  we  have  consiclerefl.  the  nsf'  ol  iVaiiH' 
databases  has  no  direct  impact  on  the  efficacy  of  VSzV  procedures.  In  fact,  with  tlu* 
advent  of  ('LOS — especially  the  CLOS  Metaobject  Protocol — Co.MMON  Lisi>  im|)le- 
inental  ioiis  of  advanced  features  of  frame  databases  tend  to  be  simple  and  >iiaighi- 
forward  [ll]. 

8.7  Backtracking 

Whether  Ijacktracking  presents  any  special  verification  difficulties  depeud>  mi  t  In- 
type  of  backtracking  and  the  specification.  If  the  backtracking  algorithm  to  lx-  useil 
is  specified,  there  is  no  particular  difficulty  in  verifying  that  it  has  been  implemented 
correctly.  Hut  problems  can  arise  when  the  algorithm  is  impreci.selv  specified  -  as  uc 
have  already  noted,  many  specification  languages  are  not  good  at  specifying  ((nnpli- 
cated  control  regimens — or  the  algorithm  used  in  the  implementation  is  a  ■qualita- 
tive"  optimization  of  the  s|)ecified  algorithm.  .An  example  of  the  latter  is  replacing 
nai've  dependency-directed  backtracking  by  more  sophisticated  reason  maintenance. 
In  this  ca.se.  verification  of  the  algorithm  can  involve  very  complicated  reasoning 
about  state.  The  difficulty  of  verifying  the  correctness  of  such  an  algorithm  might  be 
comparable  to  verifying  the  correctness  of  a  clever  garbage  collection  algorithm,  that 
is.  it  stre.s.ses  the  present  state-of-the-art. 

Guideline  8.16  .l/ocf  soiyhisiuattd  backtracking  techniques  are  much  hnrdtr  to  n- r- 
ijij  than  -Simpler  tfchniqiifs;  thi  m  only  if  efficiency  demands  it  or  com pnriitin  hj 
icenk  informal  re nficalion  is  sufficient. 

Even  complicated  backtracking  algorithms  can  be  validated  in  a  straightforward  fa.sh- 
ion.  however.  But.  just  as  in  the  case  of  meta-level  control  structures,  conventional 
validation  requirements  are  somewhat  inappropriate  to  cases  where  substantial  meta¬ 
level  reasoning  about  control  is  performed. 

Guideline  8.17  Alth  ough  im  [lU  me  Illations  of  advanced  backtracking  tfrhniiini>  nn 
"of  especially  difficult  to  validate  (i.e.  it  is  not  especially  difficult  to  erercise  them  m 
ways  that  provide  considerable  conjidence  that  they  perform  as  they  should),  standard 
ralulation  criteria  may  be  hard  to  .'satt.sfy.  If  such  criteria  are  externally  imposed 
on  the  development  effort,  use  the  more  primitive  techniques,  such  ns  chronological 
backtracking. 


9.  For  Documenters:  Documenting  Your  Efforts 


It  lias  often  i)een  asserted  tliat  DQD-STD-2167A  is  not  really  appropriate  for  Al  de¬ 
velopment  beraitse.  no  matter  wliat  l  he  standard  may  say.  it  is  fnndamentally  based 
on  a  waterfall  development  model,  while  .AI  development  is  usually  based  on  an  it¬ 
erative  prototyping  model.  One  of  the  principal  theses  of  this  manual  is  that  2167A 
is  perfectly  appropriate  to  .AI  development — in  fact,  that  it  provides  excellent  gnid<'- 
lines  to  help  ensure  that  development  will  produce  a  well-engineered  system  if  it 
is  understood  cis  requiring  the  development  of  certain  lifecycle  products,  rather  than 
dictating  a  particular  lifecycle  process.  (See  Section  o  for  the  arguments  that  re(|nire- 
inents.  specification,  and  ilesign  documents  are  a.s  valuable  when  prototyping  as  when 
waterfall  development  models  are  enifdoyed.)  I’he  adaptation  required  to  fit  2167A  lo 
.AI  development  is  essentially  the  same  as  that  to  fit  it  to  Boehm's  Spiral  .Model  i(d- 
ihe  requirements  definition,  system  sjjecilication.  system  design.  (Mr.,  must  be  treated 
as  living  documents  that  are  revised  and  expanded  throughout  the  lifecycle. 

The  rea.sons  for  documenting  can  be  seen  most  easily  by  focusing  on  the  maint»Miance 
[)hase  of  the  lifecycle  (although  a  change  in  personnel  during  development  provides 
nearly  as  good  an  illustration).  The  software  wa.s  written  to  solve  some  problem  bv 
performing  its  task.  When  the  problem  changes,  as  problems  have  a  tendency  to 
do.  the  software  must  be  changed  as  well.  If  no  formal  record  of  requirements  was 
generated,  there  is  no  documentary  guidance  available  when  attempting  to  decide 
whether  making  a  change  will  interfere  with  the  system's  functioning.  Wi)]  the  change 
alfect  essential  features  of  the  implementation,  or  only  accidents?  If  essential  features 
are  affected,  are  they  affected  in  a  significant  way?  The  requirements  statement 
sliould  answer  these  questions  by  saying  what  the  system  must  do  in  order  to  solve 
the  problem.  What  the  system  actually  does,  how  it  does  it.  how  we  determine  that  it 
does  what  its  supposed  to  do.  and  so  on,  are  irrelevant  to  answering  these  c|ueslions; 
this  document  says  what  counts  as  a  solution,  hot  which  solution  was  implemente^d. 
Therefore,  the  system  specification,  the  system  design  documents,  verification  traces 
and  validation  suites,  and  so  on,  are  irrelevant  to  the  requirements  statement. 

Guideline  9.1  The  requirements  statement  ran  and  should  read  ns  if  it  were  written 
prior  to  the  other  documents,  as  it  cs  m  the.  waterfall  model,  even  if  it  is  the  la.-'t 
doriimenl  nctualljj  completed. 

.Moreover,  all  other  project  activities — specifying  the  system,  designing  it,  and  so  on — 
are  influenced  by  the  staff’s  current  best  understanding  of  the  system  requirements. 
For  example,  the  system  specifier  must  have  some  guiding  idea  of  what  the  require¬ 
ments  are.  for  decisions  on  what  the  system  anil  tio  are  determined  by  the  specifier''^ 
understanding  of  what  it  mu.st  do:  the  specifier’s  job  is  to  choose  and  describe  the 
specification  that  will  be  (or  has  been)  implemented  from  among  those  that  satisfv 
the  rec|uirements.  Even  if  the  requirements  are  not  well  understood  at  the  time  si)ec- 
ification  commences — the  most  common  case  in  .AI  system  development — it  is  w('ll 


worth  writing  the  best  current  understanding  of  the  ie(|uirements  down,  so  that  there 
will  be  a  record  of  the  basis  of  the  specification.  And.  unless  the  development  staff 
consists  of  a  single  person,  efficiency  dematids  that  the  specification  be  written  down, 
to  ensure  a  common  nnclerstancling  among  the  staff.  And  since  the  first  project  ac¬ 
tivity  is  an  attempt  to  understand  the  [)roi)lem  aiul  possible  approaches  to  solving 
it,  it  makes  sense  that  the  first  document  produced  should  be  a  record  the  results  of 
that  activity. 

Guideline  9.2  Tin  first  draft  of  thf  rf  quirenif^  nls  staleinf  nt  should  be  tuntfeii  at  the 
beginning  of  the  project,  as  a  basis  for  all  other  hferyclt  activities. 


But  these  same  arguments  can  i)e  applied,  mutatis  mutandis,  to  the  system  specifica¬ 
tion;  the  purpose  of  the  specification  is  to  descril)e  the  system  as  a  "black  box"  —it 
flescri!)es  the  functionality  of  the  system,  its  external  interfaces,  and  so  on — so  none 
of  tlie  information  in  documents  that  are  produced  later  in  waterfall  developments  is 
relevant:  all  other  activities — design,  acceptance  test  definition,  and  so  on,  excepting 
only  rec{uirements  definition — depend  on  the  specification;  it  is  well  worth  document¬ 
ing  the  best  current  approximation  to  the  full  final  specification,  both  as  a  record 
of  presuppositions  of  other  activities  atul  to  help  ensure  a  common  understanding 
among  the  project  staff. 

Guideline  9.3  The  system  specification  can  and  .■should  read  as  if  it  were  written 
after  the  requirements  statement,  and  before  most  code  development  and  the  system 
specification  should  be  updated  throughout  the  development  Hfe-cycle.  .4  first  draft  of 
the  specification  should  be  written  at  that  time,  ns  a  basis  for  all  subsequent  lifecycle 
activities. 

The  general  conclusion  should  be  clear  at  this  point.  Each  document  is  intended  to 
serve  a  certain  purpose  in  subsequent  development  and  maintenance.  The  ordering 
of  these  documents  imposed  by  2167A  is  really  determined  by  the  purpose  of  the 
documents,  even  though  the  ordering  is  that  actually  employed  in  waterfall  devel¬ 
opments.  (This  is  no  accident,  of  course.  The  waterfall  model  was  suggested  by  an 
examination  of  the  "natural”  ordering  of  the  processes — you  can't  specify  a  system 
unless  you  know  the  requirements,  you  can't  riesign  the  system  without  knowing  tl.e 
specification,  and  so  on — without  regard  to  the  fact  that  even  the  final  stages  of  sys¬ 
tem  validation  can  reveal  additional  requirements  on  the  system  which  entail  revision 
of  every  lifecycle  product.)  Guideline  9.3  can  be  generalized  to: 

Guideline  9.4  Each  documenl  .should  be  produced  in  initial  draft  form,  in  the  order 
called  for  in  D0D-STD-2167A.  and  should  be  updated  throughout  the  development  life¬ 
cycle.  Every  document  should,  in  final  form,  rend  ns  if  it  had  been  the  product  of  a 
waterfall  development  ejfort. 


Jointly,  the  docuineuts  serve  as  a  sort  of  rational  reconstruction  of  the  development 
effort,  a  description  of  the  order  in  which  activities  would  have  occurred  if  oid\’  we 
had  known  then  what  we  know  now. 


10.  Notes  on  Testing 


Testing  AI  sottware  is  not  radically  different  from  testing  any  otlier  type  of  soil  ware. 
The  same  general  rules  apply:  test  on  typical  inputs,  test  around  (.rlretnal  raluf.s.  lest 
t.rhoiistirdy  (e.y..  exercising  nil  statements,  exercising  all  hranchts)  if  feasible,  and 
o  on.  Set  r ion  8  contains  comments  about  the  feasibility  of  strong  validation  as  a 
function  of  programming  techniques.  This  .section  covers  the  few  differences  between 
testing  .\1  software  A:  testing  other  .software. 

First,  it  should  be  noted  that  e.xtensive  testing — from  the  subunit  to  the  svsteni 
level — usually  plays  a  central  role  in  AI  development  efforts.  Debugging  consists  of 
testing  in  the  conte.xt  of  an  error,  followed  by  changes  to  the  code  in  [)reparation  lor 
retc^sting.  .\1  programmers,  especially  LiSP  programmers,  become  very  sophist  icateil 
at  debugging  during  development,  and  good  .AI  programming  environments  proc  icle 
sophisticated  debugging  tools.  The  principal  difference,  from  the  viewjxjint  of  V.VW 
between  testing  during  debugging  and  validation  testing  is  that  the  former  is  much  h^ss 
formal.  Debugging  typically  uses  ad  hoc  tests  rather  than  a  test  suite  derived  from  a 
specification,  and  eschews  any  recording  of  the  results.  ( However,  a  number  of  studies 
have  been  made  of  heuristics  for  debugging  .AI  programs.)  Thus,  one  stratc'gy  for 
encouraging  .AI  programmers  to  employ  more  comprehensive  and  systematic  feasting 
strategies  is  to  create  tools  that  make  validation  look  more  like  debugging.  The  .ASP 
tool  described  in  .Appendix  .A  is  one  example  of  such  a  tool. 

Second,  there  is  a  very  important  practical  question  that  has  not  been  addressed  vet , 
While  developing  statements  of  requirements  and  specifications  may  be  desirable  from 
the  standpoint  of  \'Sc\ .  many  .AI  development  efforts  proceed  to  develop  code  basi>d 
on  an  informal  understanding  of  the  problem,  together  with  feedback  from  users  on 
a  series  of  prototype  systems.  At  the  conclusion  of  the  ilevelopment  effort,  tin'  only 
documentation  of  the  system  is  the  source  code  and  some  sort  of  user's  manual.  How- 
can  validation  proceed  under  these  conditions?  The  most  promising  technicpies  for 
improving  testing  in  the  absence  of  a  good  specification  are  automatic  generation  uf 
test  suites  and  so-called  "program  mutation"  techniques. 

Classically,  automatic  test  suite  generation  has  been  based  upon  structural  analN'is 
of  the  program,  since  the  program  is  the  only  sufficiently  formalized  representation  of 
the  intended  behavior.  If  other  lifecycle  products  are  sufficiently  formalized,  resis  c  <ui 
be  generated  from  them  as  well.  What  the  tests  reveal  about  the  software  depends 
on  the  lifecycle  product  from  which  they  were  derived. 

•  Tests  derived  from  requirements  contribute  directly  to  validation  (provide<l 
there  is  good  reason  to  believe  the  requirements  have  been  stated  accurateiv). 

•  Tests  derived  from  the  specification  contribute  directly  to  demonstrating  cor¬ 
rectness  of  the  implementation,  and  hence  indirectly  to  validation. 


•  Tests  derived  from  the  system  design  contribute  directly  to  verification  of  th<' 
implementation. 

•  Tests  derived  from  the  code  it.self  can  tleinonstrate  that  tlie  implementation  is 
robust,  that  it  is  free  of  minor  "tyimgraphical”  errors,  and  so  on. 

.\11  these  sorts  of  It'sting  are  desirable,  and  are  mutually  complementary. 

.\  good  example  of  deriving  tests  from  code  is  path  analysis  testing,  a  method  whose 
scope  and  limits  are  well  understood  [‘idj.  But  only  in  special  cases  can  an  auto¬ 
matically  generaterl  test  suite  guarantee  that  errors  are  detectable;  restrictions  on 
the  sort  of  error  sought  [S]  or  on  the  sort  of  program  being  tested  [21]  are  ref|nired. 
The  consensus  of  tlie  community — as  reflected  in.  e.g..  the  OSI  (.'onformance  I'esrinti 
.Methodology  and  Framework  test  case  selection  guidelines — is  that  expert  judgmeni 
and  experience  are  rec|uire(l  to  supplement  informal  heuristics. 

In  cases  where  an  appropriate  formal  specification  is  available,  fully  automatic  test 
suite  generation  may  well  l)e  feasible.  For  example,  communication  protocols  are 
often  specified  using  finite  state  automata  or  Petri  nets.  For  simple  protocols,  such  as 
TCP’s  "three  way  handshake"  tor  connection  establishment,  path  testing  teclini({ues 
supplemented  by  a  formalization  of  the  OSI  heuristics  can  be  used  to  mechanicall\ 
generate  a  reasonable  conformance  test  suite. 

Program  mutation  techniques  [10].  on  the  other  hand,  can  be  usefully  applied  in 
most  projects.  Given  only  relatively  weak  restrictions  on  the  program  [20].  mutation 
testing  can  reveal  all  "typographical"  errors  in  the  program.  Such  errors  are  usually 
among  the  most  frequently  made  and  difficult  to  detect:  good  programmers  gener¬ 
ally  understand  the  algorithms  well  and  design  correct  implementations,  but  are  not 
immune  to  simple  coding  errors,  such  as  being  off-by-one  on  a  loop  exit  condition. 
Developing  automated  support  tor  mutation  testing  is  clearly  feasible,  based  on  the 
analogy  to  genetic  algorithms,  although  no  attempts  to  do  .so  have  been  documented 
in  the  literature. 
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Part  III 


APPENDICES 


A.  ASP  Manual 


A.l  Introduction 

ASP  is  an  ;u  roiuny  tor  '\-\  Sottwarr*  Ptauuer.”  It,  is  a  software  tool  to  !><-*  used  by 
|)rogrammers  and  software  test  personnel  for  planning,  organizing,  and  executing  the 
debugging,  testing,  verification,  and  validation  of  software  systems. 


A.1.1  Motivation 

.\SP  is  designed  to  reduce  the  problems  involved  in  validating  software  that  changes. 
Many  modern  software  systems  have  very  <  omple.x  reciuirements  that  cannot  be  fully 
defined  in  advance.  These  systems  are  frequently  developed  with  a  rapirl  prototyping 
methodology,  and  they  require  extensive  software  maintenance  anti  enhancements 
even  after  they  become  operational.  Traditional  testing,  verification,  and  valirlation 
methods  assume  that  changes  to  the  software  are  infrequent  and  carefully  controlled 
l)ecause  changes  require  costly  retesting,  reverification,  and  revalitlation  each  time 
the  software  does  change. 

.Much  .\I  software  must  adapt  to  evolving  requirements,  and  this  adaptability  makes 
.-\I  software  difficult  to  validate.  .ASP  is_a  software  tool  that  reduces  the  cost  and  im¬ 
proves  the  effectiveness  of  retesting,  reverification,  and  revalidation  for  .\[  .software — 
or  for  any  other  frequently  changing  software. 

.4.1.2  ASP  as  a  Software  Tool 

.\SP  allows  programmers  and  software  testers  to  specify  and  selectively  control  ilie 
execution  of  software  test  routines.  The  test  routines  are  written  as  executable 
specifications-as  predicates  expressed  either  in  the  programming  language  in  which 
the  software  is  written  or  in  a  compatible  e.xecutable  specification  language.  These 
executable  specifications  are  stored  separately  from  the  code,  and  .ASP  associates 
them  with  appropriate  test  points  in  the  .software  implementation. 

1  hese  tests  or  executable  specifications  are  called  verifications,  and  .A.SP  makes  it  easy 
for  a  programmer  to  define  these  v'erifications  early  in  the  software  development  life 
c\  cle.  These  verifications  are  often  more  stable  than  the  object  software.  By  defining 
the  verification  early,  the  programmer  can  use  them  for  debugging  the  software  and 
also  make  them  available  to  the  personnel  involved  in  the  testing,  documentation, 
verification,  and  validation  phases  of  software  development. 

.ASP  gives  the  programmer  full  control  over  when  the  verifications  are  executed.  Since 
execution  of  the  verifications  is  fre((uently  lime  consuming.  .\SP  includes  a  declara¬ 
tive  language  for  specifying  which  verifications  are  to  be  executed  under  what  cirenm- 


stances.  These  declarative  specifications  of  what  verifications  are  to  be  e.xecuted  are 
called  validations.  A  validation  is  a  plan  For  controlling  the  execution  of  the  object 
software  while  conditionally  executing  verifications  that  monitor,  lest,  and  verify  the 
software  as  it  is  executing. 

Executions  of  these  validations  or  software  plans  are  useful  in  all  of  the  following 
roles: 


External  Debugger  -  .\SP  can  help  finrl  bugs  without  perturbing  the  code  being 
tested. 

Procedural  Debugger  -  ('omple.x  debugging  scenarios  can  be  easily  devised  in  the 
Software  Plan  by  specifying  the  conditions  under  which  arlditional  verifications 
are  to  be  executed. 

Validation  -  Validation  suites  are  organized  in  the  Software  Plan. 

Verification  -  Traditional  correctness  assertions  in  the  code  are  replaced  by  tests  in 
the  Software  Plan  outside  of  the  code. 

Code  Instrumentation  -  The  progratnincr  can  specify  in  the  Software  Plan  ways 
of  visualizing  how  the  code  is  performing  and  what  it  is  doing  in  various  cir¬ 
cumstances. 

.\SP  is  an  integrated  tool  that  supports  debugging,  testing,  verification,  and  valida¬ 
tion.  Part  of  the  idea  being  supported  by  .ASP  is  that  it  is  useful  for  programmers  to 
write  verifications  during  the  early  stages  of  software  develo|>ment  because  these  ver- 
ilications  can  then  be  reused  throughout  the  software  lite  cvcle.  I  liese  verifications 
should  not  involve  modifying  the  code  that  directly  implements  required  software 
functionality:  and.  for  performance  reasons,  these  verifications  should  be  executed  se¬ 
lectively  to  meet  the  different  goals  of  debugging,  testing,  validation,  and  operational 
execution. 

The  .ASP  software  plans  give  the  programmer  control  over  the  execution  of  software 
verifications  without  changing  the  software  being  tested  or  instrumented.  .ASP  does 
not  require  recompilation  before  executing  a  different  validation  or  software  plan, 
and  .A.SP  tries  to  shorten  rather  than  lengthen  the  programmer's  cycle  of  executing, 
debugging,  modifying,  and  re-e.xecutiiig  the  software. 

■A.SP  is  designed  so  dependencies  on  the  programming  lanaguage  in  which  the  object 
software  is  written  are  isolated.  The  current  .A.SP  tool  works  for  [uograms  written  in 
Common  Lisp  and  handles  executable  specifications  written  in  either  Common  Lisp 
or  REFINE.  It  will  be  relatively  easy  to  extend  .-A.SP  to  work  with  software  written 
in  other  languages  that  allow  function  calls  to  be  intercepted  and  redirected  without 
recompilation.  .More  effort  will  be  required  to  make  .A.SP  work  efficiently  for  languages 
like  C  and  .Ada  where  function  calls  do  not  invoKe  a  level  of  indirection. 


A.  1.3  A  Software  Planning  Methodology 


ASP  can  be  used  with  auiy  software  development  methodology.  It  is  intended  to  re¬ 
duce  the  validation  problems  associated  with  evolving  software,  and  this  subsection 
describes  a  general  software  methodology  that  is  appropriate  for  the  kinds  of  evolv¬ 
ing  software  systems  which  ASP  was  designed  to  support.  This  software  planning 
methodology  is  depicted  in  Figure  A-1. 


Figurw  A-1.  Dav^oping  Evolving  Softwan 
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In  the  early  stages  of  software  development,  an  initial  version  of  the  software  require¬ 
ments  should  be  defined.  Then  these  requirements,  the  software  specification,  and  the 
software  code  are  evolved  concurrently — with  the  software  plan  used  to  interrelated 
the  evolving  specification  amd  the  evolving  code. 

Rarely  is  a  software  implemented  in  one  phase  from  the  software  requirements.  This 
is  especially  true  of  large  or  AI  software  systems.  It  is  naturad  to  develop  it  in 
bits  and  pieces  over  time  combining  smaller  pieces  into  lairger  ones  or  stubbing  out 
smaller  pieces  and  adding  them  in  progressive  steps.  With  the  software  plamning 
methodology  supported  by  ASP,  one  tries  to  tighten  the  focus  on  this  process  by 
having  the  development  of  these  bits  amd  pieces  driven  by  partial  specifications.  In 
Figure  A-1  a.  partial  specification  is  depicted  by  a  PS  in  an  ovad. 

A  partial  specification  is  a  statement  of  truth  about  the  results  of  some  partial  compu¬ 
tation  of  a  software  implementation.  The  collection  of  adl  pairtiad  specification?*  in  this 
methodology  represents  the  current  software  requirement.  When  all  of  the  partial 
specifications  in  this  collection  are  true  we  say  that  the  current  software  requirement 
has  been  satisfied  and  we  are  ready  to  go  on  to  the  next  phase  of  development.  For 
example,  suppose  the  current  software  requirement  is  that  we  construct  a  program 
that  builds  a  table  of  the  first  n  prime  numbers.  We  could  create  three  partial  speci¬ 
fications; 

1.  A  table  exists  with  n  entrys. 

2.  Every  entry  in  the  table  is  a  prime  number. 

3.  The  n  entries  are  the  first  n  prime  numbers. 

When  ail  three  of  these  partial  specifications  are  true  then  we  have  achieved  the 
current  software  requirement.  Partial  specifications  2  and  3  could  have  been  stated 
as  one  partial  specification  but  it  may  have  been  eaisier  to  test  for  them  as  two  weaker 
statements.  In  Figure  A-1  this  is  depicted  as  a  PS  splitting  off  into  two  PSs.  For  the 
same  reasons  that  it  is  easier  to  break  prograuns  into  smaller  pieces  to  debug  them, 
it  is  easier  to  break  program  specifications  into  smaller  pieces  to  test  them. 

The  Software  Plan  in  Figure  A-1  ties  these  partial  specifications  to  the  Software 
Code  and  a  decision  can  be  made  by  the  Softwaire  Plan  that  the  current  software 
requirement  has  been  met.  With  ASP,  using  the  prime  number  example,  you  have 
semi-automated  this  decision  by  displaying  the  table  cind  visually  inspecting  the  table 
to  make  sure  that  every  number  is  a  prime  number.  But  you  could  also  have  written 
a  predicate  function  that  computes  that  every  number  in  the  table  is  in  fact  a  prime 
number.  We  call  such  a  predicate  function  cin  executable  specification.  In  many  cases 
it  is  actually  easier  to  write  an  executable  specification  than  it  is  to  visually  verify. 
In  very  complex  software  it  is  perhaps  the  only  way  to  verify.  The  art  of  using  this 
software  planning  methodology  rests  in  being  able  to  write  a  minimum  number  of 
executable  specifications  such  that  to  a  large  extent  the  software  verifies  itself. 


69 


A.1.4  Using  ASP 


To  use  ASP  one  must  supply  the  shaded  boxes  in  Figure  A-2,  that  is,  the  boxes 
labeled  ASP  Software  Plan  and  Verifications. 


Host  Software 


Figun  A‘2:  ASP  ComponBnta 

The  box  labeled  Software  Implementation  represents  the  software  implementa¬ 
tion  before  ASP  is  introduced.  To  use  ASP  nothing  in  the  software  implementation 
needs  to  change.  All  code  in  the  box  labeled  Host  Software  is  written  in  the  host 
language.  For  exiimple,  if  your  software  implementation  is  coded  in  Common  Lisp 
then  the  verifications  would  be  coded  in  Common  Lisp.  Verifications  are  usueilly  just 
simple  predicate  functions.  In  the  last  section  we  described  em  executable  specifica¬ 
tion  which  could  be  represented  as  such  a  predicate.  In  general  an  ASP  verification 
is  any  function  that  supports  verifying  the  results  of  computations  in  the  software 
implementation. 

The  box  labeled  ASP  Software  Plan  represents  a  text  file  that  you  configure.  How 
to  write  a  Software  Plan  will  become  clear  in  the  following  Find  Word  Example. 
Within  the  software  plan  you  specify  how  you  expect  the  software  to  perform.  You 
do  this  by  building  condition- action  trees  in  the  Software  Plan.  At  the  top  level  these 
condition-action  trees  are  called  validations.  Within  the  Software  Plan  are  symbolic 
references  to  program  objects  in  the  software  implementation  and  verifications.  Since 
all  of  the  references  are  symbolic,  new  implementations  and  verifications  can  be  loaded 
dynamically,  and  ASP  can  do  the  right  thing  without  recompiling  the  Software  Plan. 

We  are  now  ready  to  talk  about  the  box  labeled  ASP  tool.  To  use  ASP  the  first 
thing  that  you  do  is  write  a  Software  Plan  which  attempts  to  demonstrate  the  results 
of  the  current  phase  of  software  development.  You  then  load  the  Software  Plan  and 
run  the  ASP  Tool.  The  ASP  Tool  recognizes  the  loaded  plan  as  the  current  Software 
Plan.  Most  of  what  the  Tool  does  is  based  on  interpreting  the  Software  Plan.  You 
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then  select  various  validation  scenarios  depending  on  the  desired  effect.  During  the 
interpretation  of  any  validation.  .\.SP  may  load  iinph'mentations  and  verifications  and 
control  execution  of  the  host  software  depending  on  conditions  in  the  validation. 

A.  1.5  Learning  by  Example 

We  use  the  method  of  teaching  l)V  e.xample.  VVe  will  first  give  a  simple  snccinci 
example  of  using  .\.SP,  then  we  will  explain  how  to  use  .ASP  in  general.  Since  this 
example  is  delivered  with  .\SP  it  might  he  helpful  lo  actually  load  its  Software  Plan 
and  interactively  follow  the  example.  .\  following  chapter  will  describe  the  complete 
semantics  of  the  Software  Plan.  Section  .\.6  ha.s  the  complete  syntax  of  the  Software 
Plan.  Examples  of  using  some  of  ASP's  more  esoteric  features  are  given  later. 
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A. 2  Find  Word  Example 


The  <’xaiui)le  Host  Software  we  use  is  callecl  Word.  It  iinplemeais  a  search 

aluorithtn  in  Common  Lisp.  Csing  this  algoriiiim  Find  Word  will  find  the  mimlx'r 
of  occurretR-es  of  a  given  word  in  some  given  text.  For  example  the  word  >enlf  iu:( 
occurs  d  time  in  the  te.xt  "  This  sf  iitencf  is  an  <.rninplf-  scntmcf  for  finfliiif]  th(  irortl 

sf  nif  ncf." 

We  will  Hrst  describe  the  algorithm  and  write  the  implementation.  Then  we  will  write' 
<1  Software  Plan  to  show  that  the  implementation  performs  the  way  that  we  expect. 
.\lt  hough  this  is  not  a  very  complex  exatnple.  it  will  illustrate  the  idea  of  writing  an 
.\.SP  Verification,  testing  the  software  with  multiple  implementations,  theti  deftugging 
the'  software  given  that  it  fails  the  test.  This  exantple  also  illnstratc's  the  importance' 
of  t'xecntable  specifications  because  in  this  case  one  .-VSP  V'erificalion  that  we  wiiic 
will  re'present  an  executable  specification. 

A. 2.1  Writing  the  Find  Word  program 

The  intuitive  approach  to  find  a  word  in  some  text  is  to  search  for  the  first  letter  of 
the  word  and  if  found  then  compare  the  second  letter  in  the  worel  with  the  ue'xt  letter 
in  the  text  and  so  on.  This  is  called  a  linear  search.  It  turns  out  that  we  can  search 
faster  than  that  by  using  a  sublinear  search  algorithm. 

What  we  do  in  a  sublinear  search  is 

1.  First  look  at  the  character  at  the  words  length  into  the  text. 

1.  If  this  character  is  the  Icist  character  in  the  word  search  ba(  kwards  in  the  text 
to  see  if  we  are  on  the  word. 

5.  [f  this  character  is  another  character  in  the  word  we  skip  ahead  in  the  text  bv 
the  same  number  of  characters  from  where  the  character  is  m  the  word  (closest 
to  the  end  of  the  word)  to  the  end  of  the  word.  .-Viul  continue  the  search. 

f.  Otherwise  we  skip  ahead  the  length  of  the  word.  Viul  continue  rlie  search. 

For  example  if  we  are  looking  for  the  word  ■’sentence''  in  the  text  above 


"This  sentence  is  am  exaunple  sentence  for  finding  the  word  sentence" 


skip  points; 


1  2  3 


First  we  go  into  the  text  to  skip  point  1.  the  length  of  the  word  “seiitence' .  Since 
there  is  an  “n"  there  we  skip  ahead  to  point  2  where  the  end  of  the  word  "sentence" 
would  be  if  its  character  "‘n"  was  in  that  position.  Since  there  is  an  "e"  we  search 
liackwards  and  di,scover  that  we  are  not  sitting  on  the  last  character  of  "sentence.’ 
so  we  skip  to  point  for  the  same  reason  that  we  skipped  to  point  2.  .Now  we  search 
l)ackward.s  and  find  a  match. 

This  skipping  effect  produces  a  faster  search.  We  use  sublinear  search  here  not  for 
its  efficiency  per  se  but  because  sublinear  search  needs  a  skip  table  set  up  lor  a  given 
word  prior  to  doing  the  search.  We  can  specify  what  the  skip  table  is  to  Icajk  like 
given  any  word.  .And  in  fact  we  write  a  predicate  function  that  tests  for  what  we 
specify.  .And  we  will  refer  to  it  in  the  .ASP  software  plan.  In  .ASP  terminology  we  call 
this  test  predicate  a  Verification. 


A. 2. 2  Find  Word  Code 


First,  we  write  the  code  called  WORD-SEARCH  to  implement  Fintl  Word  using  ''nhlinr-ai 
search.  This  code  was  adapted  I'roni  the  book  Hotv  to  Solrt  il  hij  Coiiijititfr  l)\ 
R.  (I.  Dromey.  VVe  also  write  a  top  level  function  <'alled  COUNT-WORD  that  lalls 
WORD-SEARCH  just  so  that  we  can  print  out  the  results.  For  this  eNample  it  is  udi 
important  to  understand  the  code;  just  notice  that  we  call  SETSKIPS  to  s<>i  up  ilu- 
s'kip  table. 


(defun  COUBT-WORD  (word  text) 

(let  ((n  (word-search  word  text))) 

(format  t  ""'/.The  word  "s  occurs  'a  times  in  the  text  ■*/.\’‘'a'a"  word  n 
(if  (<  (length  text)  101)  text  (subseq  text  0  99)) 

(if  (<  (length  text)  101)  ...")) 

(values) ) ) 

;  Given  a  word  and  text  find  the  occurrences  of  word  in  text  using 
;  a  sublinear  search  algorithm. 

(defun  WORD-SEARCH  (word  text) 

(let  ((wlength  (length  word))  (tlength  (length  text)) 

;;;  Set  up  skip  table. 

(skip  (SETSKIPS  word  (make-array  128))) 

(nmatches  0)) 

;  ;  ;  The  i  index  skips  through  the  text  to  the  character  nxt . 

(do  ((i  (1-  wlength)))  ((>  i  tlength)  nmatches) 

(let  ((nxt  (char-code  (aref  text  i)))) 

; ;  Use  skip  table  to  drive  search  for  pattern. 

(if  (>  (aref  skip  nxt)  0)  (setq  i  (+  i  (aref  skip  nxt))) 

;;;  Skip  table  indicates  maybe  at  end  of  word  so  match  backwards, 
(let  ((j  (1-  i)) 

(k  (-  wlength  2)) 

(match  t) ) 

(do  ()  ((or  (<  k  0)  (not  match))) 

(cond  ((eql  (aref  text  j)  (aref  word  k)) 

(decf  j)  (decf  k)) 

(t  (setq  match  nil)))) 

(when  match  (incf  nmatches)) 

; ; ;  Skip  in  the  text  to  where  the  end  of  the  word  would  be 
;  ;  ;  based  on  the  chsiracter  nxt. 

(setq  i  (-  i  (aref  skip  nxt))))))))) 


:  Set  up  the  ship  table  of  all  ascii  characters  for  the  given  word. 

(defun  SETSKIPS  (word  skip) 

(let  ((ulength  (length  word))) 

: ; ;  The  default  skip  for  all  characters  not  in  the  word  is  the  word  length 
(dotimes  (i  (length  skip))  (setf  (aref  skip  i)  wlength)) 

;;;  Otherwise  the  skip  is  determined  by  the  character's 
; ; ;  position  in  the  word. 

(do  ((j  1  (1+  j)))  ((>  j  (-  wlength  2))) 

(setf  (curef  skip  (char-code  (aref  word  j)))  (-  wlength  j  1))) 

: ; :  assign  negative  skip  to  last  character  to  differentiate  from  others 
(let  ((p  (char-code  (aref  word  (l-  wlength))))) 

(setf  (aref  skip  p)  (-  (auref  skip  p)))) 
skip) ) 


A. 2. 3  Find  Word  Trials 

To  check  our  code  we  create  two  trials  called  TRIALl  and  TRIAL2.  Actuallv  these 
trials  are  well  engineered  tor  this  example  so  that  the  Hrst  one  will  produce  the 
correct  answer  and  the  second  one  will  produce  an  incorrect  answer  (for  our  first 
implementation  of  WORD-SEARCH)  becau.se  of  two  extra  spaces  between  "This  "  and 
"sentence"'. 


(defun  TRIALl  () 

(count-word 

"sentence" 

"This  sentence  is  an  ex2unple  sentence  for  finding  the  word  sentence")) 

(defun  TRIAL2  () 

(count-word 

"sentence" 

"This  sentence  is  aui  example  sentence  for  finding  the  word  sentence")) 


A. 2. 4  Find  Word  Verification 


To  verify  that  the  skip  table  is  set  up  as  we  spe»  ified  we  create  the  test  predicate.  oi 
verihcatioii,  that  we  meiitionefl  earlier.  'I’oii  might  notice  that  SKIPTABLE-CORRECT  is 
about  the  same  .sii:e  as  the  function  SETSKIPS.  Don’t  let  this  discourage  you  against 
using  .\SP  to  do  program  verihcatioii.  This  is  apparent  becau.se  the  example  is  so 
simple.  As  the  complexity  of  an  implementation  goes  np.  the  size  of  the  \erificat ions 
goes  down  in  proportion  to  the  length  of  the  implementations  code:  conseciuentl y  the 
return  in  investment  becomes  greater. 


(defun  SKIPTABLE-CCRRECT  (word  tabxe) 

"For  every  character  in  Word  there  is  a  corresponding  entry  in  the 
skiptable  that  skips  relative  to  the  word’s  end  character  position.  The 
entry  that  corresponds  to  the  last  character  position  is  a  minus  skip.  All 
other  entries  contain  a  skip  equal  to  the  length  of  the  word." 

(let  ((predicate  t) 

(word-max  (1-  (length  word)))) 

(dotimes  (i  (length  table)) 

(let  ( (word-char-position  '-'osition  (code-char  i) 

word  : from- end  t ) ) ) 

(unless 

(if  word-char-posit ion 

(if  (=  word-char-posit ion  word-max) 
c.  inusp  (aref  table  i)) 

(=  (aref  table  i)  (-  word-max  word-char-position))) 

(=  (aref  table  i)  (length  word))) 

(setq  predicate  nil)))) 

predicate) ) 


Tf) 


A. 2. 5  Find  Word  Software  Plan 


Finally  we  write  a  Soltware  Plan  for  F'iiid  VVorri. 


(asp: specif y-plan 
tname  ’find- word 
: specifications 
’ ( (skip-table-specification 

"/systems/asp/examples/f ind-word/verify”) ) 

: implementations 

’ ( (f ind-word-sublinear-implementation 

'■/systems/a?D/examples/f ind-word/sublinear" 

'•/systems/ asp/examples/f  ind-word/trials'* ) 

(new-set skips -implement at ion 
'•/systems/asp/ examples/f ind-word/new-setskips” ) ) 

: executables  ’((triall)  (trial2)) 

: verification-points  ’((setskips  word  : output  skiptable)) 

: verifications  ’ ( (skiptable-correct)  (list)) 

: validations 
■ ((LOAD-FIND-WORD 

(( :load  f ind-word-sublinear-implementation  skip-table-specification) 

(: on-entry  : report)  (; on-exit  :report))) 

(TRIALl 

((; execute  triall))) 

(TRIALl -VERIFY 

((rexecute  triall)  (:on-entry  :report)  (:on-exit  :report) 

( ( : after  setskips ) 

((;test  (skiptable-correct  word  skiptable)) 

(: on-pass  : report)  (: on-fail  :report))))) 

(TRIALl-DEBUG 

((:execute  triall)  (:on-entry  :report)  (:on-exit  :report) 

( ( : after  setskips) 

((;test  (skiptable-correct  word  skiptable)) 

(:on-fail  : report 

((:record  (list  word  skiptable)  skiptable-snapshot) ) ) ) ) ) ) 

(TRIAL2 

((lexecute  triaLl2))) 

(LOAD-HEW-SETSKIPS 

( ( ;  load  new-setskips-implementation)  (;on-exit  -.report))))) 


Reading  the  plan  from  top  to  bottom:  Its  name  is  find-word.  It  has  one  spec¬ 
ification  called  skip-table-specification  which  refers  to  the  file  verify.cl  which 
contains  the  SKIPTABLE-CORRECTT  function.  It  heis  two  implementations.  One  called 
find-word-subl inear- implementation  which  contains  the  code  we  defined  earlier 
for  the  implementation  of  sublinear  search  and  its  trials,  .-\uother  called  new-setskips 
-implementation  we  will  use  later  to  fix  a  problem  in  the  original  code.  It  speci¬ 
fies  that  there  are  two  possible  execution  point  functions  called  triall  and  trial2. 


riiere  will  be  one  verification  point  called  setskips  whose  input  arguments  will 
get  l)t)uiRl  to  the  Plan  scoped  variable  word  and  whose  output  argument  will  get 
lioiiiu.l  to  the  l  ian  scoped  variable  skiptable.  There  is  one  verihralion  specified  calleil 
skiptable-correct  that  we  mentioned  earlier  and  another  veriliration  list  which 
is  just  tin'  commou  lisp  function  LIST.  A  test  (or  e.\'e<  u table  specification)  must  be 
a  \ei  i'i(  ,il  ion.  but  a  verification  does  not  need  to  be  a  lest,  for  e.Kaaiple  in  this  ca,se 
tht“  verilic  ation  list  is  just  u.sed  to  record  a  list  of  outputs  as  i>ne  objer  t.  Finally 
lor  t  his  example  we  choose  to  hav'^e  six  validations  called;  LQAD-FIJID-WQRD ,  TRAILl , 
TRIALl-VERIFY,  TRIALl-DEBUG ,  TRIAL2,  LOAD-NEW-SETSKIPS. 

A. 2.6  Using  the  ASP  tool 

that  we  have  defined  a  Software  Plan  for  the  Find  Word  software  we  are  readv 
to  inn  the  .ASP  tool  and  run  our  software  through  the  trails.  First  you  would  load 
the  .\.SP  svstem  (see  site  specific  RE.AD.ME  file  for  how  to  do  this),  and  then  load 
the  Find  Word  .Software  Plan. 

In  the  lisp  listener  we  evaluate  the  following 


(asp: tool) 


After  doing  this  we  will  see 


Select  ASP  activity  lor  FIND-WORD  plan: 

0  =  Exit  ASP  tool. 

1  =  Select  a  plan  validation  for  execution. 

2  =  Viev  PO  attributes. 

3  =  Change  plan  defaults . 

Enter  a  number  from  0  to  3  ->  1 


.\t  this  point  we  enter  a  1  to  Select  a  plan  validation  for  execution.  W’e  would 
then  see 


Select  and  execute  one  of  the  plan  validations: 
0  =  Execute  no  validation 

1  =  LOAD-FIND-WQRD 

2  =  TRIALl 

3  =  TRIALl-VERIFY 

4  =  TRIALl-DEBUG 

5  =  TRIAL2 

6  =  LOAD-HEW-SETSKIPS 

Enter  a  number  from  0  to  6  ->  1 


.Notice  that  tlie  names  of  the  (j  validations  that  we  specified  in  the  Software  Plan 
now  appear  as  selectable  validations.  We  will  first  choose  1  to  load  the  Find  Word 
software.  It  is  not  necessary  to  have  such  a  Load  Software  validation  in  your  software 
plan,  you  may  load  your  software  independently  of  .ASP.  But  it  is  included  here  to 
illustrate  two  things: 

1.  Loading  software  in  itself  can  be  considered  to  be  a  validation 

2.  When  you  get  involved  with  more  comi)lex  Software  Plans  that  load  man\' 
implementations  over  the  course  of  dilferent  tests,  you  frequently  want  to  have 
a  Validation  that  simply  reloads  the  base  implementation. 


After  entering  1  we  would  see  the  following  appear: 


ASP  controlling  FIND-WORD  plan.  I 

Loading :  FIND-WORD-SUB-LIMEAR-IMPLEMENTATIOM  I 


ASP  controlling  FIND-WORD  plan. 

Loaded;  FIND-WORD-SUB-LINEAR-IMPLEMENTATION 


ASP  controlling  FIND-WORD  plan. 
Loading :  SKIP-TABLE-SPECIFICATIQN 


ASP  controlling  FIND-WORD  plan. 
Loaded:  SKIP-TABLE-SPECIFICATION 


Looking  back  at  the  Software  Plan  notice  that  in  the  LOAD-FIND-WORD  Validation  wt' 
told  it  to  report  the  results  on  ent<*ring  and  exiting  the  load  proces.s  for  f  ind-word- 
sublinear-iraplementation  and  skip-table-specif  ication  and  the  above  output 
shows  that  it  did  just  that. 


(LOAD-FIND-WORD 

( ( : load  f ind-aord-sublinear-implementation  skip-table-specification) 
(: on-entry  ; report)  (:on-exit  :report))) 


At  this  point  we  will  see  the  top  level  menu  again.  Por  the  rest  of  what  follows  we 
will  not  repeat  this,  but  assume  that  you  will  know  that  you  start  from  the  top  lev('l 
menu. 


Select  ASP  activity  for  FIMD-WORD  plan; 

0  =  Exit  ASP  tool. 

1  =  Select  a  plan  validation  for  execution. 

2  =  View  PO  attributes. 

3  =  Change  plan  defaults. 

Enter  a  number  from  0  to  3  ->  1 

Select  and  execute  one  of  the  plan  validations: 
0  =  Execute  no  validation 

1  =  LOAD-FIMD-WQRD 

2  =  TRIALl 

3  =  TRIALl-VERIFY 

4  =  TRIALl-DEBUG 

5  =  TRIAL2 

6  =  LQAD-NEW-SETSKIPS 

Enter  a  number  from  0  to  6  ->  2 


This  time  we  select  2  for  the  TRIALl  validation.  Looking  at  the  Software  Plan  we  see 
tliat  this  Validation  does  nothing  more  than  just  execute  the  function  called  trial  1 
with  no  other  conditions.  It  is  if  we  ran  the  program  without  .ASP  being  around 
and  it  i)roduces  the  output  of  the  Find  Word  program  that  it  would  if  triall  were 
run  bv  itself: 


The  word  "sentence"  occurs  3  times  in  the  text 

"This  sentence  is  an  example  sentence  for  finding  the  word  sentence" 


Tin  s  i:)utput  looks  correct  and  in  tact  it  the  [)rogrammer  saw  this  he  might  assume 
that  Ids  program  is  working  fine.  But  now  we  will  try  TRIALl-VERIFY  Validation: 


Select  and  execute  one  of  the  plain  validations: 
0  =  Execute  no  validation 

1  =  LOAD-FIMD-WORD 

2  =  TRIALl 

3  =  TRIALl-VERIFY 

4  =  TRIALl -DEBUG 

5  =  TRIAL2 

6  =  LOAD-MEW-SETSKIPS 

Enter  a  number  from  0  to  6  ->  3 


I  ASP  controlling  FIND-WORD  plan.  I 

I  Entering:  TRIALl  I 


I  ASP  controlling  FIND-WORD  plan. 

I  Verification  SKIPTABLE-CORRECT  :  AFTER  point  SETSKIPS. 
I 'Result  =  •♦•FAILED*** 


The  word  "sentence"  occurs  3  times  in  the  text 

"This  sentence  is  an  example  sentence  for  finding  the  word  sentence" 


I  ASP  controlling  FIND-WORD  plan. 
I  Exiting;  TRIALl 


Tins  time  instead  of  just  the  program  output  we  see  tliat  .ASP  is  now  engaged. 


Looking  at  this  Validation  in  tlie  Software  Plan 


(TRIALl-VERIFY 

((: execute  triall)  (; on-entry  ; report)  (: on-exit  : report) 
((.after  setskips) 

((:test  (skiptable-correct  aord  skiptable)) 

(: on-pass  : report)  (; on-fail  ;report))))) 


We  see  that  wo  told  it  to  report  on  the  result  of  the  test  skiptable-correct  in  l)otli 
cases  of  a  pass  or  a  failure  of  the  tost.  In  the  case  of  our  implementation  of  I'ind 
VVorrl  it  reports  a.  failure.  1  his  looks  p(*ruliar  in  light  of  the  conect  output,  hut  wns 
intentionally  u.sed  in  this  example  to  illustrate  that  icorking  software  is  not  aluav-. 
I'aliii  OR  correct! 

Furthermore  we  now  go  on  to  show  that  by  executing  TRIAL2  the  output  can  be  made 
wrong.  TRIAL2  is  exactly  the  same  as  TRIALl  except  for  two  extra  spaces  between 
"This"  and  "sentence"  in  the  text; 


Select  and  execute  one  of  the  plan  validations: 

0  =  Execute  no  validation 

1  =  LQAD-FIND-HORD 

2  =  TRIALl 

3  =  TRIALl-VERIFY 

4  =  TRIALl-DEBUG 

5  =  TRIAL2 

6  =  LOAD-HEW-SETSKIPS 

Enter  a  number  from  0  to  6  ->  5 

The  word  "sentence"  occurs  2  times  in  the  text 

"This  sentence  is  am  example  sentence  for  finding  the  uord  sentence" 


.\  serendipitous  discovery  here  was  that  verification  and  debugging  of  programs  arc 
not  only  closely  related  but  are  part  of  a  continuum,  such  that  where  verification  cu(i> 
blends  into  debugging  and  vice  versa.  If  debugging  is  made  sophisticated  enough, 
at  some  point  in  the  continuum  debugging  and  verification  are  one  and  the  same. 
Furthermore  verification  can  support  debugging  and  vice  versa.  We  give  a  taste  of 
this  idea  in  this  Find  Word  example.  The  validation  TRIALl-DEBUG  is  pretty  much 
like  TRIALl-VERIFY 


(TRIALl-DEBUG 

((:execute  triall)  (:on-antry  ;report)  (;on-«xit  rreport) 


((:after  setskips) 

.((■.test  (skiptable-correct  word  skiptable)) 

(; on-fail  ; report 

((:record  (list  word  skiptable)  skiptable-snapshot )))))) ) 


W  hat,  is  (.lilicKmt  is  that  we  take  advantage  of  when  the  verification  fails  to  do  some 
d<'l)uggiiig.  So  given  :on-fail  we  not  only  report  the  results  of  tlie  test  but  we  recoid 
a  snapshot  of  the  skiptal)le. 

The  actual  interaction  with  the  .ASP  tool  when  selecting  this  Validation  goes  as  follows 


Select  and  execute  one  of  the  plan  validations: 
0  =  Execute  no  validation 

1  =  LQAD-FIND-WQRD 

2  =  TRIALl 

3  =  TRIALl-VERIFY 

4  =  TRIALl-DEBUG 

5  =  TRIAL2 

6  =  LOAD-(fEW-SETSKIPS 

Enter  a  number  from  0  to  6  ->  4 


I  ASP  controlling  FIMD-WORD  plan. 
I  Entering:  TRIALl 


I  ASP  controlling  FIND-WORD  plan. 

I  Verification  SKIPTABLE-CORRECT  ;  AFTER  point  SETSKIPS. 
I  Result  =  •••FAILED*** 


The  word  "sentence"  occurs  3  times  in  the  text 

"This  sentence  is  an  example  sentence  for  finding  the  word  sentence” 


i  ASP  controlling  FIND-WORD  plan. 
I  Exiting;  TRIALl 


.\t  this  point  the  results  look  the  same  as  the  TRIALl-VERIFY  because  we  patterned 
TRIALl-DEBUG  after  it.  But  since  we  included  the  :  record  given  an  :on-fail  con¬ 
dition  we  can  e.Kpect  to  see  an  attribute  value  for  SETSKIPS,  namely  a  snapshot  of  it 
when  the  test  failed. 


Select  ASP  activity  for  FIND-WORD  plan: 


0  =  Exit  ASP  tool. 

1  =  S«l«ct  a  plan  validation  for  exacution. 

2  =  Vias  PO  attributas. 

3  =  Changa  plan  daf aults . 

Entar  a  numbar  from  0  to  3  ->  2 

Salact  a  Program  Ob j act  to  viaw  its  attributas: 

0  =  Salact  nothing 
1  =  SETSKIPS 

Entar  a  numbar  from  0  to  1  ->  1 

Program  objact  SETSKIPS  has  tha  follouing  attributes 

SKIPTABLE-SMAPSHQT  is  a  SINGLE  valua  = 

( "sentanca" 

tt(8  3883888883888388888888888888883388888 
88888838888338883888888888888888888838 
3888888833888838888888818  -3  888388382838 
83488888883888)) 


hi  the  top  level  menu  we  choose  View  PO  attributes  { PO  stands  lor  Program  Ob¬ 
ject).  which  shows  that  the  program  object  SETSKIPS  now  ha^i  recorded  attributes. 
When  we  look  at  the  attributes  we  see  an  attribute  called  SKIPTABLE-SNAPSHOT  which 
has  the  value  of  the  word  given  to  SETSKIPS  and  the  actual  contents  of  the  table  when 
the  test  failed.  The  table  looks  pretty  good  for  the  word  "sentence"  (?.\'cept  for  one 
thing.  Since  this  is  an  ascii  value  indexed  table  we  see  a  correct  skip  of  4  for  the  "t" 
position  in  the  table,  but  just  before  it  in  the  ’’s"  position  we  see  an  S  anrl  expect 
to  see  a  7  since  ”s"  in  "sentence"  is  seven  characters  away  from  the  end  of  the  word 
">entence".  .\ha!  Could  it  be  that  we  have  a  problem  with  the  indexing  of  the  word 
and  especially  of  the  first  character?  Yes  indeed.  When  we  translated  the  algorithm 
from  Pascal  in  our  text  book  to  actual  code  we  forgot  that  Pascal  defaults  to  1  origin 
arrays  and  in  this  case  we  want  0  origin  arrays. 

We  correct  this  problem  with  a  new  implementation  of  SETSKIPS  and  now  have  ASP 
bring  in  this  implementation  with  a  Validation  called  LOAD-NEW-SETSKIPS. 


Select  emd  execute  one  of  the  plan  validations; 
0  =  Execute  no  validation 

1  =  LOAD-FIND- WORD 

2  =  TRIALl 

3  =  TRIALl -VERIFY 

4  =  TRIALl -DEBUG 

5  =  TRIAL2 

6  =  LOAD-NEW-SETSKIPS 

Enter  a  number  from  0  to  6  ->  6 


I  ASP  controlling  FIID-WORD  plan. 

1  Loaded:  MEW-SETSKIPS-IMPLEMEHTATIOH 


.\nd  nou'  we  retrv  TRIAL2  and  and  it  works: 


Select  and  execute  one  of  the  plan  validations: 

0  =  Execute  no  validation 

1  =  LOAD-FIHD-WORD 

2  =  TRIALI 

3  =  TRIALl-VERIFY 

4  =  TRIALI-DEBUG 

5  =  TRIAL2 

6  =  LOAD-MEW-SETSKIPS 

Enter  a  number  from  0  to  6  ->  5 

The  word  "sentence"  occurs  3  times  in  the  text 

"This  sentence  is  an  example  sentence  for  finding  the  word  sentence" 


« 


S6 


Finally,  just  as  a  d<-‘tail,  we  can  retry  TRIALl-VERIFY  and  see  rliat  it  now  pa.ssps  the 
test: 


Select  3Lnd  execute  one  of  the  plan  validations: 
0  =  Execute  no  validation 

1  =  LOAD-FIMD-WORD 

2  =  TRIALl 

3  =  TRIALl-VERIFY 

4  =  TRIALl-DEBUG 

5  =  TRIAL2 

6  =  LQAD-SEW-SETSKIPS 

Enter  a  number  from  0  to  6  ->  3 


I  ASP  controlling  FIMD-WORD  plan. 
I  Entering:  TRIALl 


I  ASP  controlling  FIMD-WORD  plan. 

I  Verification  SKIPTABLE-CQRRECT  ;  AFTER  point  SETSKIPS. 
I  Result  s  PASSED' 


The  Bord  "sentence"  occurs  3  times  in  the  text 

"This  sentence  is  an  example  sentence  for  finding  the  word  sentence" 


I  ASP  controlling  FIMD-WORD  plan. 
I  Exiting:  TRIALl 


A. 3  Complete  Semantics  of  the  Software  Plan 


Ill  this  sectioii  the  complete  semantics  of  the  Software  Plan  are  ('xplained  by  lisliinf 
the  keyword  arguments  of  the  function  specif y-plan  aitd  their  meanings.  Section 
.\.ti  contains  the  coniplete  syittax  of  tiie  Software  Plan.  Since  kevword  argumeitts  are 
Common  Lisp  keywords,  they  can  be  specified  in  any  order  in  n’lation  to  each  other 
that  the  user  linds  most  readable.  Requiring  quoting  of  these  keyword  argnmenis. 
such  as  a  quoted  list,  was  done  intentionally,  so  that  one  could  use  variables  or 
expressions  for  keyword  arguments  values  if  they  tieetled  that  fie.xibility  and  also  to 
make  it  easier  to  prograntmaticalh  generate  Software  Plans.  The  term  phin  .^ropfd 
idtntijiers  refers  to  all  of  the  symlxds  mentioned  in  the  Software  Plan  tltat  only  have 
values  within  the  scope  of  the  Software  Plan.  When  the  software  plan  is  interpret (sl 
by  ASP  conditions  in  the  software  plan  cause  .-\SP  to  arrange  for  the  special  handling 
of  program  ol)jects  in  the  implementation  code  such  that  when  those  objects  are 
invoked  randomly  within  the  implementation,  .ASP  will  get  control.  We  refer  to  this 
arrangement  as  condtl tom  d  corlo  aiul  t  he  invocation  of  such  objects  as  trKjtjenng. 

For  most  of  the  keywords  a  brief  description  suffices.  We  first  list  and  give  brief 
I lescri prions  of  ail  the  keywords.  The  keywords  that  need  more  detailed  descriptions 
will  be  expounded  later.  We  now'  list  the  specif  y-plan  keywords: 

:name  -  The  name  of  the  plan  (such  as  find-word).  This  needs  to  be  a  symbol, 
riiis  is  used  to  disringuish  one  Software  Plan  from  another  and  in  report  idcu- 
rification. 

.log- file  -  .A  string  re()resenling  the  file  specification  of  the  file  that  the  results  of 
:log  \’alidation  will  get  ap|)ended  to.  If  not  specified  the  log  file  defaults  to 
/.ASP-log  . 

:verbose-loading  -  Set  to  t  it  yon  want  the  actual  hies  being  loaded  displayed  during 
a  ;  load  action. 

;undefined-attribute- value  -  When  referencing  the  values  of  :  record  attributes, 
if  the  attribute  is  undefined  this  value  is  returned.  It  defaults  to  the  symbol 
; undefined . 

:collection-coercion-type  -  When  rofereuciiig  the  value  of  :  collect  at  tributes, 
which  are  recorded  as  sequences,  they  will  be  coerced  to  this  data  tvpe.  The 
default  value  is  list. 

:globals  -  .A  list  of  the  global  clauses  that  associates  plan  scoped  identifiers  with 
global  identifiers  of  the  host  language. 

:specifications  -  .A  list  ot  specification  clauses  that  associates  file  specification  strings 
with  plan  scoped  identifiers.  These  files  will  contain  user  defined  code  repre¬ 
senting  \'erilicat  ions. 


:implementations  •  list  of  iinpleineiitatioti  clauses  t  hat  associates  tile  specification 
strings  with  plan  scoped  irlentifiers.  These  files  will  essentially  be  the  code  of 
the  user's  software  implementation,  including  midtiple  implementations. 

:executables  -  A  list  of  e.\'ecution  claiis<‘s  t  hat  defines  how  .\SP  will  pass  control  to 
the  host  language  code. 

rverification-points  -  .A  list  of  clauses  t  hat  defiin-s  where  in  the  users  code  verifica¬ 
tions  can  lake  place. 

iverifications  -  A  list  of  clauses  that  defines  how  A.SP  will  invoke  user  definerl  \ei- 
ilications. 

:sub- validations  -  list  of  user  defined  validations.  These  are  exactly  the  same  as 
: validations  but  do  not  appear  in  the  .ASP  tool  selection  menu. 

:validations  -  .A  list  of  user  detined  validations. 

.4.3.1  Software  Plan  Constants 

(?onstaiits.  referred  to  as  plan  con.il.iiUs.  may  appear  in  the  Software  Plan,  in  certain 

positions.  They  may  be  either  an  integer  or  a  string.  For  example 


12345 

-1 

"Beginning  of  test" 


are  perndssible  plan  constants. 

.4.3.2  Plan  Scoped  Identifiers 

Several  constructs  in  a  Software  Plan  take  arguments  much  like  a  function  call.  But 
one  construct,  the  :  verifications-points  clauses  gire  rather  than  take  arguments. 
This  means  that  when  a  plan  scoped  identifier  is  included  in  the  clause,  its  value  is 
not  passed  to  the  function  named  hi  the  clause  but  instead  the  plan  scoped  identifier 
is  bound  to  the  value  of  the  actual  parameter  when  the  function  gets  triggered.  In 
the  Find  Word  example  the  : verifications-points  clause 


(setskips  word  ; output  skiptable) 


causes  tlie  plan  scoped  identifier  word  to  get  bound  to  the  setskips  function’s  first 
argument  value  whenever  setskips  happens  to  be  triggered  by  the  impU'inentation 
coile.  Similarly  the  plan  scoped  identifier  skiptable  gets  beimd  to  the  returned  value 
of  setskips.  .\ote  that  such  bindings  are  only  ready  to  l)e  made  in  the  scope  of  a 
:  before  or  :  after  condition.  Once  such  a  binding  actually  oc  curs  by  a  triggering, 
its  scope  in  the  Software  Plan  is  indefinite.  Or,  in  other  words,  the  scoped  identifier 
will  have  that  value  in  the  local  scope  of  the  plan  indefinitely  until  the  ne.xt  time  it 
is  caused  to  be  changed  as  specified  in  the  plan.  So  in  the  h'irnl  Word  example  once 
word  gets  bound  to  setskap’s  first  argument  value  that  will  be  iis  visil)le  value  in 
the  Software  Plan  until  the  next  triggering  of  setskips. 

.\11  otlu'r  occurrences  of  plan  scoped  identifiers  pass  the  value  boui\d  to  the  identifier  to 
a  spc'citied  tunction  in  the  enclosing  construct.  In  the  Finrl  Word  example  < ousidering 
I  he  (  onstrnct 


(skiptable-correct  word  skiptable) 

the  verihcatioii  (function)  skiptable-correct  gets  passed  the  current  |)laii  l)onnd 
\alues  for  the  plan  scoped  identifiers  word  and  skiptable. 

Plan  scoped  identifiers  are  also  used  to  specify  attributes  and  the  attribute’s  value 
binding  has  the  same  scope  rules  as  a  plan  scoped  identifier’s  regular  value  binding. 
In  the  Find  Word  example  considering  the  construct 

(: record  (list  word  skiptable)  skiptable-snapshot) 

I  he  plan  .scoped  identifier  skiptable-snapshot  becomes  the  attribute  that  gets  a 
value  as  a  result  of  the  :  record  action.  A  plan  scoped  identifier  can  be  both  a 
regular  value  and  an  attribute  value  at  the  same  time. 

A. 3. 3  Software  Plan  Arguments 

.\.s  mentioned  in  the  previous  section  various  constructs  can  tahe  arguments.  Follow¬ 
ing  •'ections  will  describe  the  meaning  of  the  arguments  for  the  particular  construct, 
I'liless  otherwise  stated  an  argument  specified  in  the  Software  Plan,  with  the  excep¬ 
tion  of  arguments  for  verification  points,  can  be  one  of  the  following; 

1.  .\  plan  constant  (as  defined  above) 

2.  .\  plan  scoped  identifier  (as  defined  above) 

•i.  .\  recorded  attribute  value  reference 

1.  .\  verification  form  (as  defined  below) 

')() 


:implementations  -  A  list  of  impleineiitatimi  clauses  that  associates  file  specificatioa 
strings  with  plan  scoped  identifiers.  These  files  will  essentially  be  the  code  of 
the  user's  software  implementation,  including  multiple  implementations. 

:executables  -  A  list  of  e.'cecution  clau.s<>s  that  defines  how  .\SP  will  pass  control  to 
the  host  language  code. 

;verification-points  -  A  list  ot  clauses  that  defiin's  where  in  the  users  code  verifica¬ 
tions  can  take  place. 

•.verifications  -  A  list  ol  clauses  that  defines  liow  .\SP  will  invoke  u.ser  defined  Ver¬ 
ifications. 

:sub- validations  -  list  of  user  defined  validations.  These  are  exactly  the  same  as 
: validations  but  flo  not  appear  in  the  .\.SP  tool  selection  menu. 

tvalidations  -  .-V  list  of  user  tlefined  validations. 

A. 3.1  Software  Plan  Constants 

Constants,  referred  to  as  i)lan  constants,  may  appear  in  the  .Software  Plan,  in  certain 

positions.  They  may  be  either  an  integer  or  a  string.  For  example 


12345 

-1 

"Beginning  of  test" 


are  perntissible  plan  constants. 

A. 3. 2  Plan  Scoped  Identifiers 

.Se\'Pral  constructs  in  a  Software  Plan  take  arguments  much  like  a  function  call.  But 
one  construct,  the  ;  verifications-points  clauses  gire  rather  than  take  arguments. 
I  his  means  that  when  a  plan  scoped  identifier  is  included  in  the  clause,  its  value  is 
not  passed  to  the  function  named  in  the  clause  but  instead  the  plan  scoped  identifier 
is  bound  to  the  value  of  the  actual  parameter  when  the  function  gets  triggered.  In 
the  Find  Word  example  the  : verifications-points  clause 


(setskips  word  : output  skiptable) 


In  the  case  of  a  Software  Plan  constant  the  constant’s  value  is  used  as  the  actual 
argument.  In  the  case  of  the  plan  scoped  identifier  its  Software  Plan  currently  l)oiind 
value  is  used  as  the  actual  argument. 

A  lecorded  attribute  value  has  the  form; 

( tthjtcl-tutnn  ntlnhxil e.-ide at ifier) 

wlu're  objert-annit  is  the  name  of  the  object  that  gets  the  attribute,  alt nhutt -nU  nlijif  r 
is  the  plan  scoped  identifier  that  names  the  attribute.  The  value  of  the  aitriintic 
ntlribiile-identijif-r  of  object  named  object-name  is  used  as  the  actual  argument. 

Verification  forms  are  defined  in  the  section  Software  Plan  ; verifications.  The 
value  returned  as  a  result  of  .\SP  invoking  the  verification  is  iiserl  ,,s  the  a(iiifil 
a  rgument. 


A. 3. 4  Software  Plan  :globals 
The  ;globals  value  is  a  list  of  clauses  of  the  form: 
iplan-nrg  globnl-val) 

the  global-ral  can  be  either  a  plan  constant  or  a  symbol  in  which  case  it  is  interpret ed 
as  an  identifier  of  the  host  language  that  names  a  global  variable,  plau-nrg  is  a 
plan  scoped  identifier  that  is  associated  with  the  global-ral  in  the  scope  of  the  whole 
Software  Plan.  For  example,  in 

.-globals  ’((pdb  ^person-data-base*) 

(alert  "We  are  changing  implementations  at  this  point")) 

pdb  is  a  [>lan  scoped  identifier  that  is  associated  with  the  global  variable  *person-data-base* 
and  alert  is  a  plan  scoped  identifier  that  is  associated  with  the  constant  string  "We 
are  changing  implementations  at  this  point". 

A. 3. 5  Software  Plan  :  specif ications 
The  :  specif  ications  value  is  a  list  of  clauses  of  the  form 
{ sgecificatton-name  string  string  ...) 

siiecificntion-nnrne  is  a  plan  scoped  identifier  that  names  an  implementation.  The 
implementation  is  associated  with  string  string  ...  that  are  file  specifications  of  hie-, 
in  the  host  operating  system  environment.  For  example. 


:  specifications 
'  ( (SKIP-TABLE-SPECIFICATION 

"/systems/asp/examples/f ind-word/verify") ) 


names  one  specification  called  SKIP-TABLE-SPECIFICATION  that  gets  associateii  with 
one  file  in  the  host  operating  system. 

A. 3. 6  Software  Plan  :  implementations 

The  :  implementations  valin'  is  a  list  of  clauses  of  the  form 

{irDjjlenifiitdfion-iKinie  -liuiq  ^tnng  ...) 

implf  nientation-dniiie  is  a  plan  scoped  identifier  that  names  an  implementation.  The 
implementation  is  as.sociat<'d  with  ■^trnig  string  ...  that  are  file  specifications  of  files 
in  the  host  operating  system  environment.  For  e.xample 


:  implementations 

'  ( (FIND-WORD-SUBLINEAR- IMPLEMENTATION 

''/systems/a-p/examples/f  ind-word/sublinear" 
"/systems/ asp/ examples/find-word/trials") ) 


names  one  implementation  called  FIND-WORD-SUBLINEAR-IMPLEMENTATION  that  gets 
associated  with  two  files  in  the  host  operating  system. 

A. 3. 7  Software  Plan  -.executables 

The  ;  executables  value  is  a  list  trf  clauses  of  the  form 

(  executable- name  nrg  arg  ... ) 

where  executable-name  is  the  name  of  a  function  in  the  software  implementation  ami 
arg  arg  ...  are  plan  scopetl  identifiers.  The  executable-name  is  usually  named  in  the 
:execute  action  which  results  in  a  call  to  the  implementation’s  function.  In  this  call 
the  actual  values  of  arg  arg  ...  are  pas.sed  as  parameters. 


A. 3. 8  Software  Plan  : verification-points 

The  : verification-points  value  is  a  list  of  clauses  of  the  form 

( i'f  rification-point  ary  ary  ...  :  output  output-ary  output-ary  ...) 

where  vt-  rijication-poinl  is  the  name  of  a  function  in  the  software  iniphunentation 
and  ary  ...  are  plan  scoped  identifiers  that  get  l)ound  to  the  function's  actual 
parameters  when  the  function  gets  triggered.  If  :output  is  specified  then  the  plan 
scoped  identifiers  output-ary  output-ary  ...  will  get  l)Ound  to  the  function's  output 
parameters. 

The  cerijirat ion-point  also  serves  as  an  identifier  for  the  : before,  .after  couditious. 
The  above  bindings  will  only  occur  during  the  scope  of  a  :  before  or  :  after  condii  itui 
and  during  that  condition  when  the  verification  point  gets  triggered. 

A. 3.9  Software  Plan  : verifications 
The  : verifications  value  is  a  list  of  clauses  of  the  form 
{ i'enficntion-nanif:  ary  ary  ...) 

Where  rn-ificatton-nnine  is  the  name  of  a  verification  that  the  user  writes.  The 
verification  must  be  a  predicate  function.  The  ary  ary  ...  are  plan  scoped  identifiers 
whose  values  get  passed  to  the  verification  whenever  the  verification  is  iiuoked  in  a 
Software  Plan  validation. 

\ou  may  specify  ary  ary  ...  in  the  verification  ''lause  mentioned  above  or  in  a  ver¬ 
ification  form  that  occurs  as  an  argument  in  a  validation  or  both.  The  arguments 
in  the  verification  form  will  always  override  the  arguments  in  the  verification  clause. 
Some  moflifications  to  the  Find  Word  e.xample  will  make  this  clear.  In  the  Find  Worrl 
example  we  specified; 

: verifications  W (skiptable-correct)  ...) 

(TRIALl-VERIFY  . . . 

...  (;test  (skiptable-correct  word  skiptable))) 

But  we  could  have  gotten  exactly  the  same  effect  with 

; verifications  '((skiptable-correct  word  skiptable)  ...) 

(TRIALl-VERIFY  . . . 

...  (:test  skiptable-correct)) 


Fulfherniore  if  we  had  specified 


: verifications  ’ ( (skiptable-correct  word  skiptable)  ...) 

(TRIAL! -VERIFY  . . . 

...  (:test  (skiptable-correct  new-word  new-skiptable) ) ) 

Then  the  skiptable-correct  verili<at.ioti  would  have  reeeived  parameters  ihal  were 
the  values  of  i  he  plan  scoped  identifiers  new-word  and  new-skiptable  since  the 
identifiers  in  the  verification  torin  override  the  irlentiliers  in  the  verification's  danse. 

In  general  a  verification  form  look.s  like 


rmjicntitjn-nnnif 
(  I'f: nfication-iKiiiif-  ur/j  nifj  ...) 


where  rf  rijicat ton- name  is  defined  in  a  verification  clause  in  the  ; verifications  list. 

In  the  fir.st  form  .-\SP  looks  at  the  verification  clairse  to  compute  the  verification's 
actual  arguments.  In  the  second  form  ary  arg  ...  is  used. 

A. 3. 10  Software  Plan  :  sub-validations 

The  semantics  of  :  sub- validations  are  exactly  the  same  as  the  semantics  of  :  validations. 
The  >)idy  difference  is  that  they  do  not  show  up  in  the  validations  .selection  menu  in 
the  .-\.SP  tool.  This  is  liandy  for  validations  that  '.on  want  to  invoke  from  other 
valiflations.  like  subroutines,  but  not  be  selectable. 


A. 3. 11  Software  Plan  validations 

The  .  validations  value  is  a  list  of  clause's  of  the  form 

I  rnlidation-name  action  action  ...) 


where  ralidation-name  appears  in  the  .-\SP  tool  validations  selection  menu,  ralidation- 
nanie  uiay  also  occur  any  place  that  an  action  can  occur  in  any  other  validation.  The 
effect  of  selecting  validation-name  in  the  .-\SP  tool  menu  or  invoking  it  from  another 
validation  is  simply  to  have  the  .A.SP  tool  interpret  the  actions  action  action  ...  .  The 
following  .sections  will  e.vplain  these  actions  in  more  detail. 


<)[ 


A. 3. 11.1  Action  Condition  Semantics 


The  general  schema  of  any  ASP  Software  Plan  validation  is  base<l  on  actions  and 
conditions  which  have  the  following  forms; 

((u  tion  coml/lion  condition  ...) 

{cniiilitioii  fiction  action  ...) 

What  rliis  means  is  that  given  action  ASP  will  perform  that  action  on  the  iniplenien- 
tation  code  subject  to  that  code  being  conditioned  by  condition  condition  ...  .  And 
given  the  one  instance  of  conditioned  code  coiiflilion.  .\SP  will  perform  tin'  actions 
action  action  ...  on  the  implementation  code. 

Furthermore  any  action  in  action  action  ...  has  precisely  the  form  of  I  lie  top  form  and 
any  condition  in  condition  condition  ...  has  precisely  the  form  of  tin'  l>ottom  form. 
This  recursive  delinition  implies  that  actions  can  have  conditions  which  can  have 
actions  which  can  have  conditions  ...etc.  This  gives  the  Plan  writer  the  capal)ility 
to  have  nested  actions  based  on  verification  conditions  carried  out  to  finer  and  liner 
levels  of  detail.  This  is  a  natural  paradigm  for  verifying  software. 

One  should  not  assume  that  .-\SP  performs  this  chain  of  actions  and  conditions  in  a 
sequential  control  thread  as  in  a  conventional  programming  language.  Consider  the 
following  construct: 

(action-a  ((:before  verif ication-point-1)  action-1) 

((ibefore  verif ication-point-2)  action-2) 

((ibefore  verif ication-point-3)  action-3  action-4)) 

action-2  I'onld  occur  any  number  of  times  before  action-1  depeiuling  on  when 
verif ication-point-2  and  verification-point-1  trigger  in  the  implementation 
code.  In  fact,  action-1  might  never  occur  if  verification-point-1  never  triggers. 
However,  it  verif  ication-point-3  triggers,  action-3  and  action-4  are  guaranteed 
to  occur  in  succession. 

Within  any  validation  the  only  kinds  of  top  level  entities  that  can  occur  are: 

Predefined  actions  -  actions  that  are  prerlefined  by  .\SP 

Predefined  conditions  -  conditions  that  are  predefined  by  .ASP 

Predefined  validations  -  .ASP  camted  validations  which  can  occur  anywhere  that 
an  action  can  occur 

User  defined  validations  -  l.'ser  defined  validations  whose  names  can  occur  any¬ 
where  that  an  action  can  occtir 

The  predefined  actions  conditions  and  validations  are  explained  and  itemized  in  the 
following  sections. 


A. 3. 11. 2  Predefined  Actions 


TIu'  predefined  at  t.ioiis  have  the  following  meanings; 


:load  -  Specified  specifications  or  implementations  are  loath'd  into  the  ASP  condi- 
rioiied  host  implementation. 

rexecute  -  A  specified  executable  is  control  executed  by  .ASP.  I’suallv  it  will  ha\e 
conditions  i  hat  alTect  verifications. 

rengage  -  Intended  to  be  the  same  semantics  as  rexecute  but  provides  an  escape 
mechanism  for  u.ser  intervention  or  manual  code  execution. 

rtest  -  Specified  verifications  that  act  as  predicates  are  control  executed  be  ASP. 

I’he  results  of  t  liese  Verifications  will  each  res|)oncl  to  subsequent  :  on-pass 
and  :  on-f ail  conditions. 

rrecord  -  The  value  of  the  specified  verification  is  recorded  as  the  valiu'  of  the  spec¬ 
ified  object's  attribute. 

icollect  -  The  same  meaning  as  rrecord  except  that  the  value  is  recorded  in  a 
secptence  which  becomes  the  attribute’s  value.  The  sequence  is  ordered  l^y  most 
recently  recorded  first. 

rreport  -  There  is  a  ])redehnecl  ; report  validation  that  does  a  specific  canned  thinu 
depending  on  its  enclosing  condition.  The  rreport  action  gives  th(*  user  I'otitrol 
o\er  what  is  reported  and  when  it  is  reported. 

rlog  -  rin.'re  is  a  predefined  ;  log  validation  that  does  a  si^ecific  eatined  thing  de¬ 
pending  on  its  enclosing  condition.  The  rlog  action  gi\es  the  user  r  ontrol  over 
what  is  logged  and  when  it  is  logged. 


The  form  of  the  ;  load  action  is 


I  :  load  name  name  ...) 


where  name  name  ...  are  the  names  of  implementations  or  specifications  delinei.l  by 
the  :  implementations  or  ;  specif  ications  keywords.  If  the  existing  host  imple¬ 
mentation  is  conditioned.  .ASP  maintains  the  conditions.  Multiple  implementatiotis 
can  be  introduced  this  way  by  loading  and  overlaying  same  named  functions. 

Tile  form  of  tlie  rexecute  action  is 


')b 


(  :  execute  name) 


where  namt  is  an  executable  defined  by  the  : executables  keyword.  When  inter¬ 
preted.  .ASP  will  first  condition  the  implementation  based  on  the  conditions  ot  the 
:  execute  then  call  the  liost  function  named  by  the  executable. 

The  form  of  the  : engage  action  is 
(: engage  : listener) 

where  :  listener  is  the  type  of  engagement.  Others  t  vpes  of  engagement  may  !)(' 
introduced  in  future  .\SP  releases.  When  interpreted  .ASP  will  first  condition  the 
implementation  based  on  the  conditions  of  the  : engage  then  give  control  to  the  .ASP 
Lisp  Listener.  While  in  the  .ASP  Lisp  Listener  by  evaluating  a  :c  the  user  may  gicc 
control  back  to  the  .ASP  lc:)oL  Lite  l)asic  idea  of  ;  engage  is  to  have  the  same  seniani  ics 
as  :  execute  but  without  the  execution  of  any  it.ser  code.  This  has  two  uses,  first, 
as  a  way  tor  the  user  to  inspect  his  environment  at  some  point  in  the  Software  l^lan. 
.\nd  second  as  a  way  for  the  user  to  tnanually  e.xecute  code  at  some  point  in  the 
Software  Plan  while  his  code  is  conditioned  by  the  Software  Plan.  Since  this  is  a 
difficult  concept  to  convey,  an  example  of  doing  this  Iw  extending  the  Find  Word 
example  is  given  in  a  following  section. 

The  form  of  the  :test  action  is 


(:test  venfication-form  renfiention-form  ...) 


where  re.nficntion-fonn  is  a  verification  form  as  defined  above.  If  the  :test  action 
has  any  ton-pass  or  ton-fail  conditions.  .ASP  will  invoke  the  verification  predicate 
kmcrions  based  on  venjication-form  re rificntion-form  ....  perform  the  ton-pass  ac¬ 
tions  tor  every  predicate  that  is  true,  and  perform  the  ton-fail  actions  for  eveiv 
predicate  that  is  false.  The  user  should  note  that  given  the  followingt 

((ttest  vl  v2  v3)  (ton-pass  actionl)) 

actionl  will  be  performed  for  every  predicate  of  vl  through  v3  tliat  is  true.  If  this 
is  not  desired  then  something  like 

((ttest  vl)  (ton-pass  actionl)) 

((ttest  v2)  (ton-pass  action2)) 

((ttest  v3)  (ton-pass  action3)) 

sliould  be  written. 

The  t  record  action  records  a  |)lan  scoperl  attribute  value.  It  has  one  of  two  formst 


(:  record  rnl-arg  object  atlnbute) 

(:  record  ciil-arg  attrihiitc) 

rnl-nrg  is  a  Software  Plan  argunient  as  defined  al)ove.  object  and  nttnbatf  are  plan 
scoped  identifiers.  .As  a  resnlt  of  the  ;  record  action  tlie  object  named  Iw  object  will 
acciuire  an  attribute  named  by  nttribule  with  a  value  determined  by  the  actual  value 
of  cal-arg.  This  attribute  can  then  be  lat(‘r  viewed  via  an  ASP  menu  or  used  in  other 
places  in  the  validation  or  other  validations. 

The  only  diffc'rence  in  the  second  lorm  is  that  object  does  not  appear.  In  this  case 
.\SP  uses  the  name  of  the  verification  point  in  the  last  <'nclosing  rbefore  or  rafter 
clause  as  the  object. 

The  :  collect  action  has  exactly  the  same  forrn  and  semantics  as  the  :  record  act  iou 
except  that  the  attribute  \alue  is  a  serpience  and  the  computed  : record  value  is 
made  the  first  element  of  the  se(iuence.  When  such  a  collect(d  attribute  is  u.sed  as 
a  Software  Plan  argument  i  he  actual  \alue  is  computed  by  coercing  the  se(|nence  to 
the  data  type  specified  by  the  Software  Plan  keyword  : collection-coercion-type. 
If  not  specified  this  keyword  defaults  to  the  list  data  type. 

The  :  report  and  tlog  actions  are  similar  in  what  the  :  report  and  :log  prerlefiried 
validations  do.  The  predefined  validations  do  a  specific  canned  thing  depending  on 
their  enclosing  condition,  where  as  the  actions  do  a  user  specified  thing  where  ever 
they  are  specificed  in  the  Software  Plan.  Their  form  is 

(  :  report  forwat-strnig  iml-argl  val~arg2  ...) 

I  :lo^  format-string  rnl-nrg  1  rnl-nrg-2  ...) 

where  forrnnt-stnng  is  a  Common  Lisp  type  format  control  string  and  rnl-nrgl  rnl- 
arg2  ...  are  0  or  more  Software  Plan  arguments  as  defined  above.  The  resulting  values 
of  these  arguments  are  consumed  exactly  as  the  arguments  would  be  consumed  by 
the  (  'ommon  Lisp  format  control  string.  .-\n  example  of  using  :  report  with  the  Fitul 
Word  example  is  given  in  a  following  section. 

-A. 3. 11. 3  Predefined  Conditions 

The  predefined  conditions  have  the  following  meanings: 

:on-entry  ton-exit  -  given  an  action,  before  the  action  is  performed  the  actions  of 
the  : on-entry  condition  are  performed.  :on-exit  is  similar  but  applies  after 
the  action  is  performed. 

:on-pass  ton-fail  -  The  results  of  the  last  previous  :test  verification  trigger  these 
conditions.  If  the  result  of  the  verification  predicate  is  true  the  : on-pass  con¬ 
dition's  actions  are  performed.  If  the  result  of  the  verification  predicate  is  false 
(he  ;on-fail  condition's  actions  are  j)erh)rmed. 


)S 


rbefore  rafter  -  If  a  previously  specified  : execution  causes  tlie  triggering  of  the 
: before  or  rafter  condition's  specified  verification  point,  the  condition's  a< - 
tions  are  performed  oefore  the  verification  point  in  the  case  of  a  rbefore  con- 
diiiou  and  after  the  verification  point  in  the  ca-se  of  an  rafter  condition. 

A. 3. 11. 4  Permissible  Conditions 

.A.11  conditions  can  have  any  actions  or  validations.  But  not  all  conditions  ihat  ax  i  ions 
can  have,  have  meaning.  Only  those  that  have  meaning  will  be  ptuformed.  The  om's 
that  have  meaning  are  ns  follows: 


Action  Meaningful  Conditions 


execute 

: before  ; after  : on-entry 

t  on-exit 

engage 

: before  ; after  :on-3ntry 

t  on-exit 

test 

: on-pass  ton-fail 

oad 

ton-entry  ton-exit 

record 

iu)ne 

collect 

none 

report 

none 

log 

none 

.4.3.11.5  Predefined  Validations 

The  predefined  \aliilations  are  as  follows; 

rreport  -  The  results  of  the  last  action  of  the  enclcriing  condition  are  displayed  to 
tlu'  user  ill  some  way. 

rlog  -  The  results  of  the  last  action  of  the  enclosing  condition  are  logged  to  the  log 
hie  specified  in  the  .Software  Plan. 

rabort  -  .\borts  all  nested  validations  up  to  the  current  top  le\e|  valiilation. 

A. 3. 11. 6  Using  Validations 

The  form  of  any  validation  is  simply 
rfilidiitioi) 

wliere  rtilulnt loii  is  the  name  of  a  predefined  or  a  user  defined  validation.  This  vali¬ 
dation  form  rdiuldtion  can  appear  anywhe'^e  that  an  action  can  appear.  For  example 


(: on-fail  : report  DO-MY-VALIDATION  : abort) 
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A. 4  More  Find  Word  Examples 


In  this  section  we  use  modifications  ot  tlie  Find  Word  example  to  illustrate  sonu*  ol 
ASP’s  capabilities  that  were  not  illustrated  by  the  Find  Word  program  example  itself. 

A. 4.1  An  Example  Using  the  :  report  and  :log  Actions 
In  the  Find  Word  example  vve  had  a  validation  called  TRIALl-DEBUG 


(TRIALl-DEBUG 

((; execute  triall)  (:oii-entry  :report)  (:on-exit  rreport) 

(('.after  setskips) 

((;test  (skiptable-correct  word  skiptable)) 

(:on-fail  ; report 

((:record  (list  word  skiptable)  skiptable-snapshot) ) ) ) ) ) ) 


.\fter  selecting  this  validation  with  the  .\SP  tool  vve  then  later  manually  looked  at 
the  skiptable-snapshot  attribute  value  to  visualize  what  weis  going  on.  By  using 
the  : report  action  we  could  have  specified  that  we  want  to  see  the  attribute  value 
during  the  validation. 


(TRIALl-DEBUG 

((rexecute  triall)  (: on-entry  : report)  (: on-exit  : report) 

((;after  setskips) 

((;test  (skiptable-correct  word  skiptable)) 

(: on-fail  : report 

((;record  (list  word  skiptable)  skiptable-snapshot)) 
((:report  ""’/.Word  and  Skiptable  =’V,"a"7," 

( setskips  skiptable-snapshot )))))))) 
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VVe  could  get  the  same  effect  without  having  the  plan  execute  triall  but  instead 
put  us  into  the  ASP  Lisp  Listener  from  where  we  could  execute  triall  and  trial2 
or  any  number  of  manual  executions  or  inspections.  When  we  are  lliough  wit  h  these 
manual  activities  we  enter  :c  and  the  .ASP  Lisp  Listener  put.s  us  l)ark  in  the  .ASP 
tool  where  we  started.  We  could  add  a  validation  called  simply  TRIAL-VERIFY  to 
accomplish  this. 


(TRIAL-VERIFY 

((rengage  : listener)  (: on-entry  : report)  (: on-exit  : report) 
((:after  setskips) 

((:te3t  (skiptable-correct  word  skiptable)) 

(; on-pass  : report)  (: on-fail  treport))))) 


Then  upon  selecting  the  TRIAL-VERIFY  validation  in  the  .ASP  tool  we  could  ;^et  the 
following  dialog: 


I  ASP  controlling  FIND-WORD  plan. 
I  Entering:  ASP  Listener 


ASP  Listener  with  validation  TRIAL-VERIFY  engaged. 
Enter  :c  to  continue  with  ASP  tool. 

ASP>  (triall) 


1  ASP  controlling  FIND-WORD  plan.  1 
I  Verification  SKIPTABLE-CORRECT  :  AFTER  point  SETSKIPS.  I 
I  Result  =  •••FAILED***  I 


The  word  "sentence"  occurs  3  times  in  the  text 

"This  sentence  is  an  example  sentence  for  finding  the  word  sentence" 


MIL 

ASP>  (trial2) 


I  ASP  controlling  FIND-WORD  plan. 

I  Verification  SKIPTABLE-CORRECT  :  AFTER  point  SETSKIPS. 
I  Result  =  •••FAILED*** 


The  word  "sentence"  occurs  2  times  in  the  text 

"This  sentence  is  an  example  sentence  for  finding  the  word  sentence" 


NIL 

ASP>  ;c 


I  ASP  controlling  FIND-WORD  plan. 
I  Exiting;  ASP  Listener 


Select  ASP  activity  for  FIND-WORD  plan; 

0  =  Exit  ASP  tool. 

1  =  Select  a  plcin  validation  for  execution. 

2  =  View  PO  attributes. 

3  =  Change  plsin  defaults. 

Enter  a  number  from  0  to  3  ->  0 
ASP: ;DONE 


While  in  the  .ASP  Lisp  Listener  we  evaluated  the  expression  (triall).  .\otire  that 
when  we  did  this  the  effect  was  the  same  as  selecting  TRIALl-VERIFY  except  that  we 
returned  back  to  the  .ASP  Lisp  Listener  at  which  point  we  evaluated  the  exiM-es.sion 
(trial2) . 

The  ASP  Lisp  Listener  is  the  same  as  an  ordinary  Lisp  Listener  except  tliat  the 
prompt  is  ASP>  and  when  we  evaluate  ;c,  which  stands  for  continue,  we  are  returned 
to  the  .ASP  tool. 

.Knother  use  for  the  ;  engage  action  would  be  to  include  it  without  any  conditions 
within  a  nested  set  of  actions  and  conditions  so  that  the  u.ser  could  niannailv  ins|)ecr 
his  environment  in  that  state.  For  example  given 


(actionl  (conditionl  action2  actions 

(action4  (conditions  actions 

((: engage  : listener ))))) ) 


In  rlie  case  of  condition2.  actions  would  take  place  followed  In-  tlie  .\SP  Lisp 
l.istener  getting  control. 
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A. 5  Using  ASP  with  Specification  Languages 


(  sing  ASP  with  a  specification  language  fiutlier  brings  into  focus  its  capabilities. 
Formal  specifications  play  the  part  of  executable  specifications,  testable  verifications 
are  short  logic  expressions,  (lecomposition  of  specifications  maps  into  the  idea  of 
partial  specifications,  the  before  and  after  effect  of  complex  transformations  is  aligned 
with  the  idea  of  centralizing  multi-point  tests  in  validations,  and  one  has  more  control 
over  generating  multiple  implementations. 

We  demonstrate  some  of  these  capabilities  by  giving  an  exampleof  using  .ASP  with  tlu' 
Refine  specification  language  as  the  host  language.  .At  the  same  time  we  demonstrate 
more  features  of  the  Soltware  Plan.  We  use  an  example  called  Buses  which  is  a 
toy  example  of  resource  scheduling  and  planning.  We  will  show  an  early  software 
evolution  phase  Software  Plan  that  discovers  a  constraint  fault  in  the  knowledge-base 
by  using  decomposition  of  specifications.  We  will  not  emphasize  the  interaction  with 
the  .\SP  tool  as  much  as  we  did  with  the  Find  Word  example.  The  complete  sources  of 
the  Buses  example  are  delivered  with  the  .ASP  software.  To  follow  this  example  more 
thoroughly  we  hope  that  the  user  will  load  the  Bu.ses  .software  implementation  and 
the  Buses  Software  Plan  and  apply  the  .ASP  tool  by  stepping  through  the  validations 
in  the  Software  Plan. 


A. 5.1  The  Buses  Example 

The  Buses  example  is  one  of  scheduling  resources  for  a  bus  system.  These  resources 
are  buses,  drivers,  routes  and  trips  which  are  modeled  in  a  knowledge  base  using  the 
Refine  language.  Constraints  such  as  what  types  of  buses  can  be  used  on  each  route, 
what  routes  a  driver  is  qualified  to  drive,  and  limits  on  the  amount  of  time  a  dri\er 
can  drive  during  a  day  can  be  expressed  as  assertions  using  logic  and  set-theoretic 
constructs. 

We  build  a  Buses  implementation  that  will  create  a  schedule  and  refine  the  schedule 
into  a  sequence  of  events.  We  build  executable  specifications  by  writing  verifications 
that  are  logic  expressions  in  Refine.  These  verifications  when  taken  as  a  whole  validate 
the  constraints  mentioned  above.  In  the  example  we  Invalidate  the  constraints  bv 
introducing  a  fault  in  them.  .A  Software  Plan  validation  detects  this  by  failing  one  of 
the  composite  tests.  We  then  try  a  validation  that  decomposes  that  particular  test 
into  partial  specifications  and  discover  the  fault. 

A. 5. 2  The  Buses  Implementation 

We  only  show  the  top  level  functions  for  the  buses  implementation  here.  The  Bl  SFS 
example  is  an  extension  of  a  Bus  Scheduling  example  given  by  Reasoning  Systems  in 
their  Refine  tutorial.  If  the  user  is  interested,  all  of  the  code  for  the  Buses  example 


is  delivered  with  the  ASP  software.  The  function  GENERATE-SCHEDULES  generates  all 
possible  schedules,  [t  does  most  of  its  work  by  calling  RECURS  I VE-CREATE-SCHEDULE. 


"Top-lev«l  function  that  generates  all  possible  schedules  and  returns 
the  number  found . " 

function  GENERATE-SCHEDULES  (b-w:  bus-world)  :  integer 
=  inif ialize-bus-world  (b-w); 
initialize-globals( ) ; 
recursive-create-schedule  (b-w); 
report-scheduling-done  (b-w); 

♦schedule-count* 

"Recursive  scheduling  function  for  the  bus  world.  Incorporates 
backtracking  and  finds  all  schedules." 
function  RECURSIVE-CREATE-SCHEDULE  (b-w)  ;  bus-world 
=  let  (  rts  =  the-routes  (b-w)) 

let  (  r  =  Route-with-Earliest-Uncovered-Time  (rts)) 
let  (  return-time  =  Last-bus-trip-retum-time  (r)) 

Report-new-recursion-level-and-rottte-data  (r,  return-time) ; 

(if  return-time  >=  end-time-requirement (b-w) 
then  report-schedule-found  (rts) 
else 

'/.y.  Generate  all  legal  combinations  of 
y.y,  bus  A  driver  for  next  bus-trip, 
let  (legal-bus-driver-pairs  = 

generate-legal-bus-driver-pairs  (b-w,  r,  return-time)) 

(if  empty  (legal-bus-driver-pairs) 
then  report -schedule-failure  (r) 
else 

(enumerate 

b-d:tuple(  bus,  driver)  over  legal-bus-driver-pairs 
do 

let  (new-bus-trip  =  make-structure 

( 'the-bus-trip  fl(newsymbol( ’TR) ) 
with-bus-trip-driver  C(b-d.2) 
with-bus-trip-bus  ®(b-d.l) 
on-route  ®r 

starting- at  ©return-time ’  )) 
report-new-bus-trip  (new-bus-trip) ; 

Recurs ive-Cr eat e-schedule  (b-w); 

Retract-bus-trip  (new-bus-trip) ) ) ) ; 

Report-backtracking( ) ; 
b-w 


A. 5.3  Buses  Executable  Specifications 

The  following  verifications  serve  as  executable  specifications  for  the  Buses  example. 
The  natural  language  specificatiou  that  the  ('xecutable  specification  verifies  precedes 


lUb 


each  test  verification.  Each  verification  is  based  on  resource  constraints  in  the  Buses 
object  world.  Notice  that  all  of  the  verifications  are  e.xpressed  in  Refine  as  logic 
formulas.  Also  notice  that  they  are  all  quantified  conjunctions.  This  helps  us  when 
\v('  want  to  decompose  specifications  into  partial  specifications. 


"TEST-P-l:  A  partially  completed  schedule  has  a  schedule  lor  every  route." 
function  TEST-P-l  (world:  bus-world) : boolean  = 
fa(r)  (r  in  the-routes (world)  =>  ex(s) 

(s  =  route-schedula(r) 

7.  the  times  in  s  meike  s  a  continuous  schedule  starting  at  0. 
ft  ( (def ined?(last(s) )  ft  def ined?(bus-trip-start(last(s) ) ) ) 

=>  bus-trip-3tart(last(s))  =  0.0) 
ft  fa(tl) 

(tl  in  s  => 

( (defined? (bus-trip-end(tl) )  and  def ined?(bus-trip-start(tl ) ) ) 

=>  (bus-trip-end(tl)  -  bus-trip-start (tl) 

=  trip-time(bus-trip-bus(tl) ,r) ) ) ) 

ft  fa(tl,  t2) 

((tl  in  s  ft  t2  in  s)  => 

(s  =  [. . ,t2,tl, . .]  =>  bus-trip-start (t2)  =  bus-trip-end(t 1 ) ) ) ) ) 

"TEST-C-1:  A  completed  schedule  has  a  schedule  for  every  route." 
function  TEST-C-1  (world:  bus-world) : boolean  = 
fa(r)  (r  in  the-routes(world)  =>  ex(s) 

(s  =  route-scheduls(r) 

7,  the  times  in  s  make  s  a  continuous  schedule  covering  the 

7.  whole  time  period, 

ft  bus-trip-start(last(s) )  =  0.0 

ft  bus-trip-end(f ir3t(s) )  >=  end-time-requirement (world) 
ft  fa(tl) 

(tl  in  s  => 

bus-trip-end(tl)  -  bus-trip-3tart(tl)  =  trip-time(bu3-trip-bus(t 1 ) ,r) 
ft  fa(tl,  t2) 

((tl  in  s  ft  t2  in  3)  => 

(s  =  [. . ,t2,tl, .  .]  =>  bus-trip-start (t2)  =  bus-trip-end(tl )  ^ ) ) ) ) 

"TEST-2:  No  two  buses  are  in  use  at  the  same  time, 
and  there  is  at  least  IS  minutes  for  refueling  between  uses." 
function  TEST-2  (world:  bus-world) : boolean  = 
fa(r)  (r  in  the-routes (world)  =>  ex(s) 

(s  =  route-schedule(r)  ft 

fa(tl,t2)  ((tl  in  s  ft  t2  in  s)  => 

((  def ined?(bus-trip-start(tl) )  ft  def ined?(bus-trip-end(tl ) )  ft 
defined?(bus-trip-staurc(t2))  ft  def ined?(bus-trip-end(t2) ) )  => 

(tl  '=  t2  => 

((  bus-trip-start (tl)  <  bus-trip-3tart(t2) 
ft  bus-trip-end(tl)  <=  bus-trip-start ( t2 ) )  or 

(  bus-trip-start(t2)  <  bus-trip-start ( tl )  ’ 
ft  bus-trip-ond(t2)  <=  bus-trip-start(tl ))))))) ) 
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l«t  (.schedules  =  {  route-schedule(r)  I  (r)  r  in  the-routes(Borld)>) 
fa(sl,s2)  (  (si  in  schedules  ft  s2  in  schedules)  => 

(  si  •=  s2  => 

fa(tl,t2)  (  (tl  in  si  ft  t2  in  s2)  => 

(  tl  ■=  t2  => 

((  del ined?(bus-trip-start(tl) )  ft  delined?(bus-trip-end(tl) )  ft 
defined? (bus-trip-start(t2))  ft  delined?(bus-trip-end(t2) )  ft 
delined?(bus-trip-bus(tl) )  ft  delined?(bus-trip-bus(t2))  )  => 

(bus-trip-bus(tl)  =  bus-trip-bus(t2)  => 

(bus-trip-end(tl)  +  0.2S  <  bus-trip-staa-t(t2)  or 
bus-trip-end(t2)  +  0.2S  <  bus-crip-start(tl)))))))) 

"TEST-3:  No  two  drivers  are  on  different  trips  at  the  same  time, 
cind  they  are  not  driving  more  that  8  hours  a  day." 
function  TEST~3  (world:  bus-world) : boolean  = 
fa(d)  (d  in  the-drivers(world)  => 
total-driving-time(d)  <=  8.0) 
ft 

fa(r)  (r  in  the-routes (world)  =>  ar(3) 

(s  =  route-schedule(r)  ft 

fa(tl,t2)  ((tl  in  s  ft  t2  in  s)  => 

(  tl  '=  t2  s> 

(  (  bus-trip-driver(tl)  =  btts-trip-driver(t2)  )  => 

( (bus-trip-start(tl)  >  bua-trip-end(t2)  or 
bus-trip-end(tl)  <=  bus-trip-start (t2) )))))) ) 

"TEST-4:  All  the  other  problem-specific  constraints.” 
function  TEST-4  (world:  bus-world) : boolean  = 
fa(b)  (b  in  the-buses(world)  *> 
fa(bt)  (bt  in  bus-bus-trips(b)  => 

bus-trip-driver(bt)  in  drivers-trained-for(bus-trip-route(bt ) ) 
ft  bu3-si2e(b)  in  qualified-for(bus-trip-driver(bt) ) 
ft  y2ird-of-driver(bus-trip-driver(bt) )  =  bus-yard(bus-trip-bus(bt) ) 
ft  mountain-view-restriction?(b,  bus-trip-route(b) ) 
ft  fremont-restriction?(b,  bus-trip-route(b) ) ) ) 


A. 5. 4  Buses  Constraint  Fault 

III  the  Software  Plan  the  validation  that  introduces  the  constraint  ffiiill  loads  an  imple¬ 
mentation  with  the  fault.  This  fault  is  in  the  constraint  called  BUS-DRIVER-CONSISTENCY 
which  is  a  conjunction  of  three  sub-constraints  given  a  particular  driver,  bus  and  route; 

1.  The  yard  of  the  driver  should  be  the  same  as  the  yard  of  the  bus. 

2.  The  driver  should  be  qualified  for  the  size  of  the  bus. 

d.  1  he  total  hour  limit  i.s  correct  for  the  combination  of  driver,  bus  and  refute. 


ins 


function  BUS-DRIVER-COMSISTEICY 

(d;driver,  brbus,  r:routa)  :  booleein 

*/,*/.*/,  bus-yard  (b)  =  yard-of-driver  (d)  k 
bus-siza  (b)  in  qualif iad-f or  (d)  k 
total-hour-limit-ok?  (d,  b,  r) 


For  den\onstratioa  purposes  vve  artificially  comment  out  sub-coiistiaint  nuinl»ei  1. 
This  is  equivalent  to  leaving  it  out  at  some  phase  of  the  software  evolution. 

A. 5. 5  Buses  Partial  Executable  Specifications 

Because  of  the  fault.  TEST-4  will  fail,  so  we  decompose  TEST-4  into  partial  spe(  ifica- 
tions  and  create  verifications  TEST-4-1  through  TEST-4-5: 


function  TEST-4- 1  (world;  bus-world) rboolaan  = 
fa(b)  (b  in  tha-busas(world)  => 
fa(bt)  (bt  in  bus-bua-trips(b)  => 

bus-trip-drivar(bt)  in  drivars-trainad-for(bus-trip-routa(bt) ) ) ) 

function  TEST-4-2  (world;  bus-world) ;boolaan  * 
fa(b)  (b  in  ths-busas( world)  => 
fa(bt)  (bt  in  bus-bus-trips (b)  => 

bus-size(b)  in  qualif ied-for(bus-trip-driver(bt) )) ) 

function  TEST-4-3  (world;  bus-world) ; boolean  = 
fa(b)  (b  in  tha-busas(world)  => 
fa(bt)  (bt  in  bus-bus-trips (b)  => 

y2u:d-of-drivar(bus-trip-driver(bt) )  =  bus-y2u:d(bus-trip-bus(bt) ) ) ) 

function  TEST-4-4  (world;  bus-world) ; boolean  = 
fa(b)  (b  in  tha-busas (world)  => 
fa(bt)  (bt  in  bua-bus-trips(b)  => 
mountain-viaw-rastriction?(b,  bus-trip-routo(b) ) ) ) 

function  TEST-4-5  (world;  bus-world) ; boolean  = 
fa(b)  (b  in  tha-busas (world)  => 
fa(bt)  (bt  in  bus-bu3-trips(b)  => 
freiBont-rsstriction?(b,  bus-trip-route(b) ) )) 


A. 5. 6  Buses  Software  Plan 

We  now  show  the  Buses  Software  Blau  that  refers  to  the  implementations  and  speci¬ 
fications  that  we  discus.sefl  alxne. 
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(asp: sp«cily-plan 
:nane  ’busss 
: spscilications 

’ ( (busas-sp«cilication  "/ systems/ asp/ examples/buses/vnv-spec" ) 

(buses-1 iner-specilication  "/systems/asp/exaaples/buses/vnv-spec-sub") ) 

: implementations 

'  ( (buses-base-implementation  "/systems/ asp/ examples/buses/load-base" 

" / sy St  ems/ asp/ examples /buses/ imp 1 “ ) 

(buses-faulty-implementation  ”/systems/asp/examples/buses/plant-bug") ) 
tglobals  '((h2  *world2*)) 

;executables  ’ ( (generate-schedules  a2)) 

: verification-points  ’ ( (report-new-bus-trip)  (report-schedule-found) ) 

: verifications  '((test-p-1  w2)  (test-c-l  m2) 

(test-2  m2)  (test-3  m2)  (test-4  m2) 

(teat-4-1  m2)  (teat-4-2  m2)  (test-4-3  m2)  (te8t-4-4  m2) 
(test-4-S  m2)  (1+)  (>)) 

: sub-validations 

■ ( (SHOW-MO-REPORT-SCHEDULE-FOUKD 

((:record  0  report-schedule-found  count))) 

(LIMIT-REPORT-SCHEDULE-FOUMD 

((;record  (1+  (report-schedule-found  count))  report-schedule-found  count)) 
((:test  (>  (report-schedule-found  count)  4))  (;on-pass  :abort)))) 

: validations 
’((LOAD-BUSES 

((:load  buses-base-implementation  buses-specification) 

(; on-entry  ; report)  (: on-exit  :report))) 

(QRDIHARY-RUM 

((:execute  generate-schedulos) ) ) 

(ORDINARY-TEST-RUH 

SHOW-MO-REPORT-SCHEDULE-FOUIID 

((:execute  generate-schedulos)  (; on-entry  : report)  (;on-exit  : report) 
((:beforo  report-noM-bus-trip) 

((;test  test-p-1  test-2  test-3  teat-4) 

(; on-pass  : report)  (; on-fail  : report  : abort))) 

((;before  roport-schedule-found) 

LIMIT-REPORT-SCHEDULE-FOUMD 

((.'test  tost-c-1  test-2  test-3  te3t-4) 

(: on-pass  : report)  (:on-fail  : report  : abort))))) 
(ORDIMARY-TEST-RUM-WITH-SUSPECT 
(( :load  buses-faulty-implementation) 

(: on-entry  : report)  (: on-exit  : report)) 

QRDIHARY-TEST-RUM) 

( FIMD-SUSPECT-TEST-RUI 

( ( .'load  buses-faulty-implementation  buses-f  iner-specif  ication) ) 

((rexecute  generate-schedules) 

((: before  report-noM-bus-trip) 

((;test  tost-p-1  test-2  test-3)  (: on-fail  : report)) 

( ( : test  test-4) 

(ton-pass  : report) 

( ton-fail  : report) 

(: on-fail  ((.test  te3t-4-l  test-4-2  test-4-3  test-4-4) 


(: on-pass  : report)  (: on-fail  ; report  :abort)))))) 

( ( : before  report-schedule-found) 

((:test  test-c-l  test-2  tast-3)  (:on-fail  ireport)) 

((:test  test-4) 

( : on-pass  : report ) 

(: on-fail  : report 

((:te3t  test-4-1  test-4-2  test-4-3  test-4-4) 

(; on-pass  import)  (: on-fail  : report  : abort ))))))) ) 


.\l  this  phase  of  the  software  evolution  we  have  five  validations: 

1.  LOAD-BUSES 

2.  ORDINARY-RUN 

■I  ORDINARY-TEST-RUN 

4.  ORDINARY-TEST-RUN-WITH-SUSPECT 

5.  FIND-SUSPECT-TEST-RUN 

LOAD-BUSES  simply  loads  the  base  implementation. 

ORDINARY-RUN  simply  e.xecutes  GENERA TE-SCHEDULES.  .\SP  will  pa.ss  it  the  value  of 
rl\e  plan  scoped  identifier  w2  which  is  associated  in  the  Software  Plan  with  the  refine 
’.71  liable  *world2*  which  points  to  the  Buses  world  knowledge-ba.se.  By  selecting 
ilii.'i  validation  with  the  .ASP  tool  you  will  see  the  output  that  GENERATE-SCHEDULES 
produces  without  any  conditioning  and  output  by  .\SP. 

ORD I  NARY -TEST- RUN  does  the  same  thing  as  ORDINARY-RUN  except  that  it  condi¬ 
tions  the  implementation  code  before  the  verification  point  report-new-bus-trip 
h\-  testing  the  verifications  test-p-1,  test-2,  test-3,  and  test-4.  If  any  one 
(d’  these  tests  pass  it  will  simply  report  that  it  passed,  if  any  one  fails  it  will  re¬ 
port  that  it  fails  and  abort  the  validation  ORDINARY-TEST-RUN.  Simultaneously  it 
icuiditions  the  code  before  report -schedule-found.  Since  it  was  discovered  that 
report -schedule-found  triggers  a  huge  number  of  times.  ORDINARY-TEST-RUN  in- 
.okes  the  validation  LIMIT-REPORT-SCHEDULE-FOUND  which  limits  the  number  of  trig¬ 
gers  to  0  then  aborts. 

When  you  select  ORDINARY-TEST-RUN  you  will  see  that  all  tests  pass.  So  we  select 
ORDINARY-TEST-RUN-WITH-SUSPECT.  Notice  that  this  validation  simply  loads  the  con¬ 
straint  fault  implementation  that  we  discussed  above  and  performs  the  ORDINARY-TEST 
-RUN  validation.  This  is  common;  we  want  to  lest  the  code  the  same  way  we  did  before, 
hut  with  a  different  implementation  of  the  same  function.  Now  with  ORDINARY-TEST-RUN- 
WITH-SUSPECT  we  will  see  failures  being  reported  and  in  particular  test-4  which  will 
raii.se  .-VSP  to  output: 

ill 


I  ASP  controlling  BUSES  plan. 

I  Varilication  TEST-4  :  BEFORE  point  REPQRT-BEW-BUS-TRIP. 
I  Result  =  ♦**FAILED*** 


Filially  we  select  FIND-SUSPECT-TEST-RUN  which  will  make  use  of  our  partial  specifi¬ 
cations  we  (liscu.ssecl  above  by  loading  the  specification  buses-f  iner-specif  ication. 
In  this  validation  we  see  a  bit  more  coniple.x  nested  testing  going  on.  Given  that 
test-4  fails.  .ASP  is  directed  to  test  the  partial  e.'cecutable  specifications  of  test-4 
namely  test-4-1  through  test-4-4.  This  eventually  results  in  a  triggering  that 
produces  the  .ASP  output: 


I  ASP  controlling  BUSES  plem. 

I  Verification  TEST-4-3  ;  BEFORE  point  REPORT-MEW- BUS -TRIP. 
I  Result  =  ♦••FAILED*** 


I  ASP  controlling  BUSES  plan.  I 
I  Controlled  aborting  of  validation  I 
1  FIMD-SUSPECT-TEST-RUM  ...  I 


These  results  pin  down  the  constraint  fault  and  end  this  phase  of  software  evolution. 
The  cnrrem  validations  and  verifications  will  be  used  over  and  over  again  for  future 
phases. 


A. 6  Software  Plan  Complete  Syntax 


The  syntax  for  an  ASP  Software  Plan  is  expressed  here  in  BNF.  The  usual  B.\'F 
notations  are  used:  The  production  symbol  is  :  angle  brackets  <>  describe  non¬ 

terminals,  a  liar  I  indicates  alternatives,  curly  brackets  followed  by  an  asterisk  {}* 
indicates  0  or  more  occurrences  of  the  construct  inside  of  the  curly  brackets,  and 
anything  else  is  a  terminal  symbol. 

The  Production  tree  starts  at  that  described  under  TOP  LEVEL  and  continues  wit  h  t  hat 
described  under  B.N’F  sections  SPECIFICATIONS,  IMPLEMENTATIONS,  EXECUTABLES. 
VERIFICATION  POINTS.  VERIFICATIONS,  and  VALIDATIONS.  At  the  lowest  level  of  i  he 
VALIDATIONS  part  t)f  the  tree  are  ACTIONS  and  CONDITIONS.  .All  sections  will  reler  to 
non-terminals  described  utuler  the  PARAMETERS  and  ATTRIBUTES  sections. 

•Some  terminal  symbols  include  the  cpiote  ’  symbol.  They  are  used  for  top  le\r'| 
quoted  lists.  This  was  done  intentionally  to  make  it  easy  for  more  advanced  work 
involved  with  programmatically  generating  .Software  Plans  as  lists  and  aijplyiug  the 
function  SPECIFY-PLAN  to  tho.se  lists. 

TOP  LEVEL 

<plan-specil ication>  ( specif y-plan  {<plzm-attribute>}*) 

<plan-attribute> 

:  :  =  <plan-nane-spec>  I  clog-f-ile-speO  I  <verbose-loa<ling-spec>  I 

<undef ined-attribute-value-speO  I  <collection-coercion-type-spec>  I 
<globals-spec>  I  <3pecifications-spec>  I  <ijnplementations-spec>  I 
<executables-spec>  I  <verif ication-points-speO  I 

<verif ications-speO  I  <sub-validations-spec>  I  Cvalidations-speo 
cplain-name-speO  :nanie  ’<symbol> 

<log-file-spec>  ; log-file  <striog> 

<verbose-loading-spec>  ; :=  : verbose-loading  <choice-syinbol> 

<undef ined-attribute-value-speO  : :=  : undefined-attribute- value  <host-val> 
<collection-coercion-type-spec>  ::=  : collection-coercion-type  '<symbol> 
<globals-speo  ::=  iglobals  ’ ({<plan-arg>  <global-val>>*) 

PARAMETERS 

<global-val>  : ;=  <host-var>  I  <plan-const2uit> 

<plan-par>  ;;=  <pl2ui-au:g>  I  <recorded-val>  I  <plan-constant> 

<plan-constant>  ::=  <string>  I  <integer> 

<plan-arg>  ::=  <syinbol> 

<host-var>  ::=  <symbol> 

<host-val>  :;=  <symbol>  I  <3tring>  I  <integer> 

<choice-syinbol>  ::=  t  I  nil 

ATTRIBUTES 

<recorded-val>  ::=  (<object-najne>  <attribute-naine>) 

<record-syntax>  ::=  <record-syntax-l>  I  <record-syntax-2> 


<record-syntax-l>  ::=  <recording-val>  <attribute-naitt«> 

<r«cord-syntax-2>  ::=  <recording-val>  <obj«ct-naiii«>  <attribute-najne> 
<recording-val>  <verif ication-forni>  I  <plaii-par> 

<object-naiBe>  <syiBbol>  I  <v«rilicatioii-point-spec> 

<attribute-naa«>  ;;=  <syabol> 

SPECIFICATIONS 

<3p«cilications-spec>  :;=  ; specif icat ions  ’ ({<specification-f onB>}’*) 
<specif ication-forni>  ::=  (<specif icatioii-naiBe>  ■{<string>>*) 

<specif  ication-naine>  :  :=  <synibol> 

IMPLEMENTATIONS 

<implementations-spec>  ;:=  :  implementations  ’  (•{<implementation-form>}-*) 
<implementation-forai>  .•:=  (<implementation-najBe>  {<string>}*) 
<implementation-name>  <symbol> 

EXECUTABLES 

<executables-spec>  : : executables  ’ ({executable-form}*) 
<exacutable-form>  (<executable-name>  {<plan-par>>*) 

<exacutable-naffle>  <syrabol> 


VERIFICATION  POINTS 

<verif ication-points-speO 

; verification-points  ’ ({<verif ication-point-fonn>}*) 

<verif ication-point-fonn> 

; :=  ’ (<verif ication-point-name>  <verif ication-point-arg-spec>) 
<verif ication-point-arg-3pec> 

; ;=  {<point-arg>}*  I  {<point-arg>}*  ; output  {<point-2u:g>}* 
<verif icati''n-point-name>  ::=  <symbol> 

<point-arg>  : ;=  <symbol> 

VERIFICATIONS 

<verif ications-speo  ;:=  : verifications  ’ ({<verif ication-form>}*) 
<verification-f orm>  ;;=  ’ (<verif ication-name>  {<plan-par>}*) 

<verif ication-naiBa>  ::=  <symbol> 

VALIDATIONS 

<sub-validation3-spec>  ::=  : sub-validations  * ({<validation-fonn>>*) 
<validations-spec>  ::=  ;validation3  ’ ({<validation-fonii>>*) 
<validation-form>  ::=  (<validation-name>  {<action>>*) 
<validation-naine>  ;;=  <3ymbol> 

<predef ined-validation>  :;=  ; report  I  :log  I  : abort 


ACTIONS 


<action>  <vad.idation-reierence>  i  (<action-form>  {<conciition>}*) 
<validation-ref«rence>  :;=  <validation-najne>  I  <predef ined-validation> 
<action-fonn> 

: ;=  <load-action>  I  <execute-action>  I  <engag9-action>  I  <test-action>  I 
<record-action>  I  <collect-actioii>  1  <report-action>  I  <log-action> 
<load-action>  (:load  <load-ionii>) 

<load-torm>  ;:=  <3p«ciiication-name>  I  <implementation-nani9> 
<executo-action>  ::=  (; execute  <executable-name>) 

<engage-action>  : :=  (: engage  ; listener) 

<test-action>  (itest  {<verif ication-name>}*) 

<test-3pec>  <verif ication-naine>  I  (<verif ication-naine>  {<plan-par>}* ) 

<record-action>  :;=  (:record  <recard-syntax>) 

<collect-action>  ::=  (:collect  <record-syntax> ) 

<report-action>  (:report  <string>  {<plam-pcLr>}*) 

<log-action>  (:log  <3tring>  {<platn-pax>>*) 

CONDITIONS 

<ccnaition>  : :=  (<condition-fonn>  {<action>}*) 

<condition-fonB>  ::=  <predef ined-condition>  I  <verif ication-point-speO 
<predel ined-condition>  :on-pass  I  ;on-fail  I  :on-entry  I  ;on-exit 

<ver:f ication-point-spec> 

.:=  (<condition-naK4  '  <verif ication-point-najne>) 

<condition-name>  ::=  : before  I  ; after 


B.  Definitions  : —  Terms  and  Abbreviations 


B.l  General  Acronyms 


A  DP 
ADS 
AFB 
ATD 
ATP 
ATR 
(’(' 

(TR 

CDR 

CDRL 

CEO 

CEE 

CFI 

CFG 

CFS 

Cl 

CM 

CMP 

COI 

COO 

CI’P 

CRISD 

COTS 

CQAE 

CRC 

(  SC 

CSC! 

CSOM 

CSP 

rsSR 

CSC 

DC  A  A 

DCS 

DDCMP 

DES 

DEAR 

DIA 


Automatic  Data  Processing 
Advanced  Decision  Systems 
Air  Force  Base 
Acceptance  Test  Description 
Acceptance/ Accreditation  Test  Plan 
Acceptance  Test  Report 
Command  and  Control  Division 
Configuration  Control  Board 
Critical  Design  Review 
Contract  Data  Rec[uirements  List 
Chief  Executive  Officer 
Customer  Furnished  Equipment 
Customer  Furnished  Information 
Computer  Facilities  Group 
Customer  Furnished  Softvvar' 

Configuration  Item 
Configuration  Management 
Configuration  Management  Plan 
Commaait\  of  Interest 
Chief  Operating  Officer 
Cost  Performar.^e  Review 

Computer  Resources  Integrated  Support  Document 

Commercially  available.  Off-the-Shelf 

y'hief  Quality  .Assurance  Evaluator 

Cyclic  Redundancy  Check 

('omputer  Software  Components 

(.’omputer  Software  Configuration  Item 

Computer  System  Operator's  Manual 

Communications  Support  Processor 

Cost  Schedule  Status  Report 

Computer  Software  Cnit 

Defense  Contracting  .Audit  .Agency- 

Defense  Communications  System 

Digital  Data  Communications  .Message  Protocol 

Data  Encryption  Standard 

Defense  Federal  .Acquisition  Regulation 

Defense  Intelligence  .Agency 


DID 

Data  Item  Description 

DIS 

Daily  Intelligence  Summary 

DLR 

Direct  Labor  Rates 

DoD 

Department  of  Defense 

DSC  IS 

Daily  (LS.  Space  Command  Intelligence  Summary  Message 

’  DTG 

Date  Time  Group 

ECP 

Engineering  Change  Proposal 

EPROM 

Erasable-Programmable  Read-Only  Memory 

FAR 

Federal  Acquisition  Regulation 

FCA 

Functional  Configuration  Audit 

FCCM 

Facilities  Capital  Cost  of  Money 

FDMP 

Full  Duplex  Message  Protocol 

FOC 

Final  Operational  Capability 

FQT 

Formal  Qualification  Test 

FSM 

Firmware  Support  Manual 

GFE 

Government  Furnished  Equipment 

GFI 

Gov-ernment  Furnished  Information 

GFS 

Government  Furnished  Software 

G<Sj  A 

General  N:  Administrative 

HMI 

Human  Machine  Interface 

HOL 

Higher  order  Language 

HQ 

Headquarters 

HWCI 

Hardware  Configuration  Item 

IC'D 

Interface  Control  Document 

ICWG 

Interface  Control  Working  Group 

IDD 

Interface  Design  Document 

IOC 

Initial  Operational  Capability 

I/O 

Input  and/or  Output 

IP 

Internet  Protocol 

IRS 

Interface  Requirements  Specification 

ISS 

Intelligence  Support  Systems 

ir 

Imagery  Division 

I  WAV 

Independent  Verification  and  Validation 

12 

Intelligence  Data  Handling  Systems  Communications,  \ersion2 

JSR 

Job  Status  Report 

LAN 

Local  Area  .Network 

LLCSC 

Lower-level  computer  software  component 

LOE 

Level-of- Effort 

MCG 

.Mapping,  Charting  and  Geodesy 

MCCS 

Mission-Critical  Computer  System 

MIL 

.Military 

1 17 

MLDT 

Mean  Logistics  Delay  Time 

MMI 

Man-  Mach  ine  Interface 

MTBF 

Mean  Time  Between  Failnies 

.\[TBSF 

Mean  Time  lietween  Software  Faults 

MTBSH 

Mean  Time  Between  Software  Halts 

MTTR 

Mean  Time  to  Repair 

NDS 

Non- Development  Software 

NFS 

Network  f  ile  System 

News 

Network  extensible  Window  System 

NLP 

Natural  Language  Processing 

OA 

Operational  Availability 

OCD 

Operational  Concept  Document 

OCR 

Optical  Character  Reader 

OM 

Operator's  Manual 

OS 

Operating  System 

PC  A 

Physical  Cunliguration  Audit 

PDR 

Preliminary  Design  Review 

PID 

Process  ID 

PM 

Program  Manager 

PQM 

Program  Quality  Manager 

PRC 

Program  Review  Coordinator 

PROM 

Programmable  Read-Otdy  Memory 

PRT 

Program  Review  Team 

QA 

Quality  Assurance 

QAC 

Quality  Assurance  Committee 

QAM 

Quality  Assurance  Manager 

RCS 

Revision  (.'ontrol  System 

RFP 

Request  for  Proposal 

RGB 

Red,  Green.  Blue 

ROM 

Read-Only  Memory 

RPC 

Remote  Procedure  Call 

RTMS 

Real  Time  Message  System 

R.^D 

Research  and  Development 

SA 

System  Administrator 

SAC 

Strategic  Air  Command 

SAD 

Situation  Assessment  Division 

SCI 

Sensitive  Compartmented  Information 

SC  IF 

Special  Compartmented  Intelligence  Facility 

SCN 

Specification  Change  Notice 

SDD 

Software  Design  Document 

SDF 

Software  Development  Folder 

SDL 

Software  Development  Library 

SDP 

Software  Development  Plan 

SDR 

Software  Design  Review  or  Software  Deficiency  Report 

SDRL 

Subcontract  Deliverables  R<‘quiremeiits  List 

SER 

Software  Engineering  Report 

SI 

System  Integrators 

SO 

Security  Officer 

SOW 

Statement  of  Work 

SPM 

Software  Programmer’s  Manual 

SQEP 

Software  Quality  Evaluation  Plan 

SQPP 

Software  Quality  Program  Plan 

SPS 

Software  Product  Specification 

SRR 

System  Reciuirernents  Review 

SRS 

System  Requirements  Specification 

SSDD 

System/Segment  Design  Document 

sss 

System/ Segment  Specification 

STD 

Standard 

STP 

System  Test  Plan 

STR 

System  Test  Report 

SE 

Superuser 

SUM 

Software  User's  Manual 

sw 

Software 

SW’CI 

Software  Configuration  Item 

TBD 

To  Be  Determined 

TCP 

Transmission  Control  Protocol 

TCP/IP 

Internet  protocol  suite 

TELNET 

Telecommunications  Network  Protocol 

ITM 

Technical  Interchange  Meeting 

TOMP 

Ta^k  Order  Management  Plan 

TQM 

Total  Quality  Management 

TRR 

Test  Readiness  Review 

CDF 

Unit  Development  Folder 

IT 

L’ser  Interface 

CM 

L'ser’s  Manual 

CPS 

Uninterruptible  Power  Supply 

\DD 

Version  Description  Document 

XTcV 

\"erification  and  Validation 

WBS 

Work  Break-Down  Structure 

VV^'SIWYG 

What  You  See  Is  What  V'ou  Get  (pronounced:  Whiz-E-Wig) 

B.2  Definitions 


•  ASCII.  Standaicl  alphanunicnc  character  s('t  tor  computers. 

•  Authentication.  Determination  by  the  Government  that  specification  con¬ 
tent  is  acceptable. 

•  Background.  .A  non-realtime  message  hardcopy  document  entered  through 
the  optical  character  readers,  magnetic  tape,  or  by  hand. 

•  Baseline.  .A  configuration  identification  flocument  or  a  set  ol'  such  docu¬ 
ments  formally  designated  and  fixed  at  a  specific  time  during  a  CTs  life  cycle. 
Baselines,  plus  approved  changes  from  those  Ijaselines,  constitute  the  current 
configuration  identification.  For  'ilfiT.A  configuration  management  technicpies 
there  are  three  baselines,  as  follows: 

1.  Functional  baseline.  The  initial  approved  functional  configuration  iden¬ 
tification. 

2.  Allocated  baseline.  The  initial  approved  allocated  configuration  identi¬ 
fication. 

3.  Product  baseline.  The  initial  approvetl  or  conditionally  approved  prod¬ 
uct  configuration  identification. 

However,  for  iterative  prototyping  programs  the  functional  and  allocated  re- 
cjuirements  are  not  identified  until  late  in  the  development  process.  The  use  of 
a  test  or  development  baseline  in  place  of  the  functional  and  allocated  baselines, 
allows  for  the  system  evolution.  The  two  stanflar<l  baselines  that  ADS  utilizes 
are: 

1.  Test  baseline.  The  test  baseline  will  be  establishetl  as  segments  are 
released  from  each  task  group. 

2.  Product  baseline.  The  product  ba.seline  will  be  establiaherl  at  the  suc¬ 
cessful  completion  of  contract  requirements/deliveries. 

•  Baseline  Management,  .\pplication  of  technical  and  administrative  direc¬ 
tion  to  designate  the  documents  which  formally  ideulifv  and  establish  the  initial 
configuration  identification  at  specific  times  during  its  life  cycle. 

•  Certification.  .A  process  which  may  be  incremental,  by  which  a  contractor 
provides  objective  evidence  to  the  contracting  agency  that  an  item  satisfies  its 
specified  requirements. 

•  Computer  data  definition.  .\  statement  of  the  characteristics  of  the  basic 
elements  ol  information  operated  upon  by  harilware  in  responding  to  computer 
instructions.  These  characteristics  may  include,  but  are  not  limited  to.  type, 
range,  structure,  and  value. 


Computer  hardware.  Devices  capable  of  accepting  and  storing  computer 
data,  executing  a  systematic  sequence  of  operations  on  computer  data  or  i)io- 
ducing  control  outputs.  Such  devices  can  perform  substantial  iuterpretatit)n. 
computation,  communication,  control,  or  other  logical  functions. 

Computer  resources.  The  totality  of  computer  hardware,  software,  persou- 
iiel.  documentation,  supplies,  and  services  applied  to  a  given  effort. 

Computer  software.  A  combination  of  a.ssociated  computer  instriu  tious  and 
computer  data  definitions  required  to  enable  the  computer  hartlware  to  perfoi  iii 
computational  or  control  functions. 

Computer  Software  Component  (CSC).  distinct  part  of  a  computer 
software  configuration  item  (CS(’I).  CSCs  may  be  further  decomposc'd  into 
other  (.’SC's  and  (^omputer  Software  I’nits  ((.’SC's). 

Computer  Software  Configuration  Item  (CSCI).  configuration  item 
for  computer  software. 

Computer  Software  documentation.  Technical  data  or  infortnation.  in¬ 
cluding  computer  listings  and  printouts,  which  documents  the  reciuiretTients. 
design,  or  details  of  computer  software,  explains  the  capabilities  and  limita¬ 
tions  of  the  software,  or  provides  operating  instructions  for  using  or  supporting 
computer  software  during  the  software's  operational  life. 

Computer  Software  Unit  (CSU).  An  element  specified  in  the  design  of  a 
(Computer  Software  C'’omponent  (CSC)  that  is  separately  testable. 

Configuration,  The  functional  and/or  physical  characteristics  of  luird- 
ware/software  as  set  forth  in  technical  documentation  and  achieved  in  a  product. 

Configuration  control.  The  systematic  evaluation,  coordination.  ap[>io\nl. 
disapproval,  and  implementation  of  all  approved  changes  in  the  configuration 
of  a  Cl  after  formal  establishment  of  its  configuration  identification. 

Configuration  Identification.  The  current  approved  or  conditionally  ap¬ 
proved  technical  documentation  for  a  configuration  item  as  set  forth  in  <p<'(  ih- 
cations.  drawings  and  associated  lists,  and  documents. 

Configuration  Item  Number  (CIN).  .A  CIN  is  a  permanent  number  as¬ 
signed  by  the  configuration  manager  to  identify  a  configuration  item.  The  CIN 
is  composed  of  alpha-numeric  characters. 

Configuration  Item  (Cl).  .\n  aggregation  of  hardware /software,  or  any  of 
its  discrete  portions,  which  satisfies  an  end  use  function  and  is  designated  for 
configuration  management. 


Configuration  Mangement.  A  discipline  applying  tecluiiral  and  adminis- 
tiative  direction  to  ( 1 )  identify  and  document  the  functional  and  physical  char¬ 
acteristics  of  a  configuration  items,  (2)  control  changes  to  fliose  characteristics, 
and  (  ?)  record  and  report  change  processing  and  implementation  status. 

Configuration  status  accounting.  The  recording  and  reporting  of  the  in¬ 
formation  that  is  needed  to  manage  configuration  effectively,  iii<  hiding  a  listing 
of  the  approved  configuration  identification,  the  status  of  |)roposed  ( liaiig<‘s  to 
( onfignration.  and  the  implementation  status  of  approved  changes. 

Cost.  The  term  ■cost"  means  cost. 

1.  Non-recurring  costs.  One-time  costs  which  will  be  incurred  if  an  en¬ 
gineering  change  is  ordered  and  which  are  independent  of  the  quatititv  of 
items  changcfl.  such  as,  cost  of  redesign,  special  tooling  or  cpialificat ion, 

2.  Recurring  costs,  (.'osts  which  are  incurred  for  each  item  changed  or  foi 
each  .service  or  document  ordered. 

Critical  Design  Review  (CDR).  This  review  shall  be  conducted  for  each 
configuration  it<Mn  wheti  detail  design  is  essentially  complete.  The  purpose  of 
this  review  will  be  to  ( L)  determine  that  the  detail  design  of  the  configuration 
item  under  review  satisfies  the  performance  and  engineering  speciality  require¬ 
ments  of  the  HWCI  development  specification(s),  (2)  establish  the  detail  design 
compatibility  among  the  configuration  item  and  other  items  of  equipment,  facil¬ 
ities.  computer  software  personnel.  (3)  assess  configuration  item  risk  areas  (on 
a  technical,  cost,  and  schedule  basis).  (4)  assess  the  resulted  of  the  pioducibil- 
ity  analyses  conducted  on  system  hardware,  and  (5)  review  the  preliminary 
hardware  (irodiict  specifications.  For  CSCIs,  this  review  will  focus  on  the  d('- 
termination  of  the  acceptability  of  the  detailed  design,  performance,  and  test 
characteristics  of  the  design  solution,  and  on  the  adequacy  of  the  operation  and 
support  documents. 

Critical  item.  .-Kn  item  within  a  Cl  which,  because  of  special  engineering  or 
logistic  considerations,  requires  an  approved  specification  to  establish  technical 
or  inventory  control  at  the  component  level. 

Deficiencies.  Deficiencies  consist  of  two  type:  (1)  conditions  or  charac  teris¬ 
tics  in  any  hardware/software  which  are  not  in  compliance  with  specified  con¬ 
figuration.  or  (2)  inadequate  (or  erroneous)  configuration  identification  which 
has  resulted,  or  may  result,  in  configuration  items  that  do  not  fulfill  approved 
operational  requirements. 

Developmental  Configuration.  The  contractor's  software  and  a.ssociated 
technical  documentation  that  defines  the  evolving  configuration  of  a  CSCl  dur¬ 
ing  development.  It  is  under  the  development  contractor's  configuration  con¬ 
trol  and  describes  tbe  s^fivvare  design  and  implementation.  The  Developmental 


Configuration  for  a  CSCI  consists  of  a  Software  Design  Document  and  source 
code  listings.  .Any  item  of  the  Developmental  Configuration  may  be  stored  on 
electronic  media. 

•  Domain.  The  area  of  interest  of  a  particular  program. 

•  Engineering  change.  .An  alteration  in  the  configuration  of  a  configuration 
item  or  item,  delivered,  to  l>e  delivered,  or  under  development,  after  formal 
establishment  of  its  configuration  identification. 

•  Engineering  Change  Proposal  (ECP).  .A  term  which  includes  both  a  pro¬ 
posed  engineering  change  and  the  documentation  by  which  the  change  is  de¬ 
scribed  and  suggested. 

•  Evaluation.  The  process  of  determining  whether  an  item  or  activity  meers 
specified  criteria. 

•  Firmware.  The  combination  of  a  hardware  rlevice  and  computer  instructions 
or  computer  data  that  reside  as  read-only  software  on  the  hardware  device.  The 
software  cannot  be  readily  modified  under  program  control. 

•  Formal  Qualification  Review  (FQR).  T  he  test,  inspection,  or  analytical 
process  by  which  a  group  of  configuration  items  comprising  the  system  are 
verified  to  have  met  specific  contracting  agency  contractual  requirements  (spec¬ 
ifications  or  equivalent).  This  review  does  not  apply  to  hardware  or  software 
requirements  verified  at  FC.A  for  the  individual  configuration  item. 

•  Formal  Qualification  Testing  (FQT).  .A  process  that  allows  the  contracting 
agency  to  determine  whether  a  configuration  item  complies  with  the  allocated 
1  .'quirements  for  that  item. 

•  Flmctional  area.  .A  distinct  group  of  system  performance  requirements 
which,  together  with  all  other  such  groupings,  forms  the  next  lower  level  break¬ 
down  of  the  system  on  the  basis  of  function. 

•  Functional  characteristics.  Quantitative  performance,  operating,  and  lo¬ 
gistic  parameters  and  their  respective  tolerances.  Functional  characteristics 
include  all  performance  parameters,  such  as  range,  speed,  lethality,  reliabdity. 
maintainability,  and  safety. 

•  Functional  Configuration  Audit  (FCA).  The  formal  examination  of  func¬ 
tional  characteristics’  test  data  for  a  configuration  item,  prior  to  acceptance,  to 
verify  that  the  item  has  achieved  the  performance  specified  in  its  functional  or 
allocated  configuration  identification. 

•  Hardware  Configuration  Item  (HWCI).  configuration  item  for  hard¬ 


ware. 


Independent  Verification  and  Validation  (IVAcV).  Verification  and  val¬ 
idation  performed  by  a  contractor  or  independent  group  that  is  not  responsible 
for  developing  the  product  or  performing  rg  w  activity  being  evaluated.  IV.^:V 
is  an  activity  that  is  conducted  separately  from  the  software  development  ac¬ 
tivities. 

Interface  Control.  Interface  control  comprises  the  delineation  of  the  proce¬ 
dures  and  documentation,  both  administrative  and  technical,  contractually  nec¬ 
essary  for  identification  of  functional  and  |)hysical  characteristics  between  two  or 
more  configuration  items  which  are  provided  by  different  contractors/Coverninent 
agencies,  and  the  resolution  of  the  problem  thereto. 

Interface  Control  Working  Group  (ICWG).  For  programs  which  encom¬ 
pass  a  system/configuration  item  design  cycle,  an  ICWG  normally  is  establislu'd 
to  control  interface  activity  between  contractors  or  agencies,  including  resolu¬ 
tion  of  interface  problems  and  documentation  of  interface  agreements. 

MMI.  Term  used  to  describe  the  interface  between  the  user,  the  computer,  and 
the  program.  Terms  iiicUule:  man-machine  interface,  human-machine  interface, 
user-interface. 

Non-development  software  (NDS).  Software  that  is  not  required  to  be 
delivered  by  the  contract. 

Operator.  In  te.xt'management.  a  term  that  describes  the  connection  between 
a  subtopic  and  a  topic  (e.g.  and.  or  not.  vole,  phrase).  For  computers,  one  who 
maintains  the  computer. 

Physical  characteristics.  Quantitative  and  qualitative  expressions  of  ma¬ 
terial  features,  such  as  composition,  dimensions,  finishes,  form.  fit.  and  their 
respective  tolerances. 

Physical  Configuration  Audit  (PCA).  technical  e.xamination  of  a  des¬ 
ignated  configuration  item  to  verify  that  the  configuration  item  ".As  Built" 
conforms  to  the  technical  documentation  which  defines  the  configuration  item. 

Preliminary  Design  Review  (PDR).  This  review  shall  be  conducted  for 
each  configuration  item  or  aggregate  of  configuration  items  to  ( I )  evaluate  the 
progress,  technical  adequacy,  and  risk  resolution  (on  a  technical,  cost,  and  sched¬ 
ule  bcisis)m  of  the  selected  design  approach.  (2)  determine  its  compatibility  with 
performance  and  engineering  speciality  requirements  of  the  IIWCI  development 
specification.  (3)  evaluate  the  degree  of  definition  and  assess  the  technical  risk 
associated  with  the  selected  manufacturing  met  hods/ processes,  and  (4)  estab¬ 
lish  the  existence  and  compatibility  of  the  physical  and  functional  interfaces 
among  the  computer  software  and  personnel.  For  CSCls,  this  review  will  focus 
on:  IT)  the  evaluation  of  the  progress,  consistency,  and  technical  adequacy  of 


the  selected  top-level  design  and  test  approach.  (2)  compatibility  between  solt- 
ware  requirements  and  preliminary  design,  and  (3)  on  the  preliminary  version 
of  the  operation  and  support  documents. 

•  Qualification. 

•  Reusable  software.  .Software  developerl  in  response  to  the  recjuiremeais  for 
one  application  I  hat  can  be  used,  in  whole  or  in  part,  to  satisfy  the  requ i remen l  s 
of  another  application. 

•  Software  development  file/folder  (SDF).  A  repository  for  a  collection  of 
material  pertinent  to  the  development  or  support  of  software,  (.'ontents  t  vpically 
include  (either  flirectly  or  by  reference)  design  considerations  and  constraints, 
ih'sign  documentation  and  data,  schedule  and  status  information,  tc^st  re(|uiic- 
ments.  test  cases,  test  procedures,  and  test  results. 

•  Software  development  library  (SDL).  controlled  collection  of  software, 
documentation,  and  associated  tools  and  procedures  u.sed  to  facilitate  the  or¬ 
derly  development  and  subsequent  support  of  software.  The  SDL  includes  the 
Developmental  Configuration  as  part  of  its  contents.  .A  software  development 
library  provides  storage  of  and  controlled  access  to  software  and  documentatiuii 
in  human-readable  form,  machine-readable  form,  or  both.  The  library  may  also 
contain  management  adat  pertinent  to  the  software  development  project. 

•  Software  engineering  environment.  The  set  of  automated  tools,  firmware 
devices,  and  hardware  necessary  to  perform  the  software  engineering  effort. 
The  automated  tools  may  include  but  are  not  limited  to  compilers,  assembler-;, 
linkers,  loaders,  operating  systems,  debuggers,  simulators,  emulators,  test  tools, 
docunientat io[i  tools,  and  data  base  management  system(s). 

•  Software  quality.  The  ability  of  a  software  product  to  satisfy  its  specified 
ref|uirements. 

•  Software  Specification  Review  (SSR).  .A  review  of  the  finalized  Computer 
Software  Configuration  Item  (CSCI)  requirements  and  operational  concept.  Hie 
SSR  is  conducted  when  CSCI  requirements  have  been  sufficiently  defineil  lo 
evaluate  the  contractor  s  responsiveness  to  and  interpretation  of  the  sy-tem. 
-egment,  or  prime  item  level  requirements.  A  successful  SSR  is  predicated 
upon  the  contracting  agency's  determination  that  the  Software  Requirements 
Specification.  Interface  Requirements  .Specification(s).  and  Operational  Concept 
Document  form  a  satisfactory  basis  for  proceeding  into  preliminary  software 
design. 

•  Specification,  .A  rlocument  intended  primarily  for  use  in  procurement,  which 
clearly  describes  the  essential  technical  requirements  for  items,  materials,  or 
services  including  the  procedures  by  which  it  will  be  determined  that  the  r('- 
qiiirements  have  been  met. 


1.  General  specification.  A  document  which  covers  the  requirements  com¬ 
mon  to  different  types,  classes,  grades  and/or  stvles  of  items  or  services. 

2.  Detail  specification.  document  which  covers  (either  within  it.self 

or  by  referencing  and  supplementing  a  general  specification)  the  complete 
requirements  for  only  one  type  of  item,  or  for  a  limited  iiuml)er  of  types,' 
classes,  etc.  of  similar  characteristics. 

System  specification.  .A  document  which  states  the  technical  and  mission 
requirements  for  a  system  as  an  entity,  allocates  requirements  to  functional 
areas  (or  configuration  items),  and  defines  the  interfaces  between  or  among 
the  functional  areas. 

4.  Development  specification.  A  document  applicable  to  an  item  below 
the  system  level  which  states  performances,  interface  and  other  technical 
requirements  in  sufficient  detail  to  permit  design,  engineering  for  service, 
use.  and  evaluation. 

*).  Product  specification.  .A  document  applicable  to  a  production  item 
below  the  system  level  which  states  item  characteristics  in  a  manner  suitable 
for  procurement,  production,  and  acceptance. 

Specification  Change  Notice  (SCN).  .\  document  used  to  propose’,  trans¬ 
mit,  and  record  changes  to  a  specification. 

Synonym.  .An  operator  that  allows  a  topic  to  be  a  synonym  of  (the  same  as  I 
another  topic. 

System.  .\  composite  of  subsystems,  assemblies  (or  sets),  skills,  and  tech- 
nic(ues  capable  or  performing  and/or  supporting  an  operational  lor  non-operational  i 
role.  .A  complete  system  includes  related  facilities,  items,  material,  services,  and 
personnel  re<iuired  for  its  operation  to  the  degree  that  it  can  be  considered  a  self- 
sufficient  item  in  its  intended  operational  (or  non-operational)  and/or  suppoit 
environment. 

System  Design  Review  (SDR).  This  review  shall  be  conducted  to  evaluate 
the  optimization,  correlation,  completeness,  and  risks  associated  with  the  allo¬ 
cated.  technical  rec[uirements.  .Also  included  is  a  summary  review  of  the  system 
engineering  process  which  produced  the  allocated  technical  requirements  and  uf 
the  manufacturing  planning  for  the  ne.xt  |)hase  of  effort. 

Subcontractor.  .A  subcontractor  is  an  indeividual.  partnership,  corporation, 
or  association,  who  (which)  contracts  with  a  contractor  to  design,  develop,  de¬ 
sign  and  manufacture,  manufacture  items,  which  are  or  were,  designed  specifi¬ 
cally  for  use  in  a  military  application. 

System  Requirements  Review  (SRR).  The  objective  of  this  review  is  to 
ascertain  the  adequacy  of  the  contractor's  efforts  in  defining  system  reciuire- 
ments.  It  vvill  be  conducted  when  a  significant  portion  of  the  system  functional 
ref|nirement.s  has  been  established. 


•  Software  support.  The  sum  ol  all  activities  that  take  place  to  eusure  tiuit 
implemented  and  fielded  software  continues  to  fully  support  the  operational 
mission  of  the  software. 

•  Software  test  environment.  .\  set  of  automated  tools,  firmware  device,  and 
liardware  necessary  to  test  software.  The  automated  tools  may  include  nbut  are 
not  limited  to  test  tools  such  as  simulation  software,  code  analyzers,  etc.  and 
may  also  include  those  tools  used  in  the  software  engineering  environment. 

•  System  Specification.  .A  system  level  requirements  specification. 

•  Technical  Report.  .A  technical  report  encompasses  the  evaluated  relevant 
facts  on  a  study  or  phase  of  a  study  of  a  particular  art,  science,  [)rofession.  or 
trade,  and  stands  as  a  permanent  official  record  in  a  formal  document.  The 
prime  purpose  of  a  technical  report  is  to  dissentinate  the  results  of  activity  and 
to  foster  the  exchange  of  information. 

•  Test  Readiness  Review  (TRR).  .A  review  conducted  for  each  CSC  I  to 
determine  whether  the  software  test  procedures  are  complete  and  to  assure  that 
the  contractor  is  prepared  for  formal  CSCI  testing.  Software  test  procedures 
are  evaluated  for  compliance  with  software  test  plans  and  descri[)l  ions,  ami 
for  adequacy  in  accomplishing  test  requirements.  .At  TRR.  the  contracting 
agency  also  reviews  the  results  of  informal  software  testing  and  any  updates 
to  the  operation  and  support  documents.  .A  successful  TRR  is  predicated  on 
the  contracting  agency's  determination  that  the  software  test  procedures  and 
informal  test  results  form  a  satisfactory  basis  for  proceeding  into  formal  CSCI 
testing. 

•  Unit.  One  complete  (  onfiguration  item. 

•  Validation.  The  process  of  evaluating  software  to  determine  compliance  wii  h 
■specified  requirements. 

•  Vendor.  .A  vendor  is  a  manufacturer  or  supplier  of  a  commercial  item. 

•  Verification.  The  process  of  evaluating  the  products  of  a  given  software 
development  activity  to  determine  correctness  and  consistency  with  respect  to 
the  products  and  standards  provirled  as  ini)Mt  to  that  activity. 

•  Version.  .An  identified  and  documented  bo<ly  of  .software.  .Modification  to  a 
version  of  software  (resulting  iii  a  new  version)  require  configuration  manage¬ 
ment  actions  by  either  the  contractor,  the  contracting  agency,  or  both. 

•  Waiver.  .A  written  authorization  to  accept  a  configuration  item  or  other 
designated  items,  which  during  piwluction  or  after  having  been  submitted  for 
inspection,  are  found  to  depart  from  specified  requirements,  but  nevertheless 
are  considered  suitable  for  u.se  "as  is*  or  after  rework  by  an  approved  method. 


•  Work  Breakdown  Structure  (WBS).  A  product-oriented  taniily  tree,  com¬ 
posed  of  hardware,  software,  services  and  other  work  tasks,  which  results  from 
project  engineering  effort  during  the  tlevelopn\ent  and  prodiu  tion  of  a  defense 
material  item,  and  which  completely  defines  the  project/prograu).  A  WBS  dis¬ 
plays  and  defines  the  product(s)  to  he  developed  or  produced  and  relates  the 
elements  of  work  to  be  accomplished  to  each  other  and  to  the  end  product . 


B.3  Document  Definitions 

•  Configuration  Management  Plan  (CMP).  The  Configuration  .Manage¬ 
ment  Plan  (CMP)  describes  the  |)rocedures  and  methods  to  be  used  for  coidig- 
uration  management  during  the  life  of  the  program.  This  include  development, 
testing,  and  installation. 

•  Computer  Resources  Integrated  Support  Document  (CRISD).  [he 

Computer  Resources  Integrated  Support  Document  (CRISD)  provides  the  in¬ 
formation  needed  to  plan  for  life  cycle  support  of  deliverable  software.  The 
CRISD  documents  the  contractor's  plans  for  transitioning  support  of  deliver¬ 
able  software  to  the  support  agency. 

•  Computer  System  Operators  Manual  (CSOM).  The  Computer  .System 
Operator's  Manual  (CSOM)  provides  information  and  detailed  procedures  for 
initiating,  operating,  monitoring,  and  shutting  down  the  computer  system  and 
for  identifying/isolating  computer  malfunctions. 

•  Firmware  Support  Manual  (FSM).  The  Firmware  Support  Manual  (F.-\.M) 
|)rovides  the  information  necessary  to  load  .software  or  data  into  firmware  com¬ 
ponents  of  a  system.  It  is  e(|ually  applicable  to  read  only  memory  (  RO.Ms).  Pro¬ 
grammable  ROMs  (PRO-Ms),  Erasable  PROMs  (EPROMs),  and  other  firmware 
devices. 

•  Interface  Control  Document  (ICD).  The  Interface  Control  Document 
(K.'D)  specifies  all  of  the  external  (other  systems)  and  internal  (between  sub¬ 
systems)  interfaces  necessary  to  ensure  proper  development  of  software  for  the 
system.  It  serves  to  document  and  control  interface  decisions. 

•  Interface  Design  Document  (IDD).  The  Interface  Design  Document  ( IDD) 
specifies  the  detailed  design  for  the  interface  between  the  CS(.'Is. 

•  Interface  Requirements  Specification  (IRS).  The  Interface  Requirements 
Specification  (IRS)  specifies  the  retiuirements  for  one  or  more  interfaces  between 
one  or  more  CSCIs  and  other  configuration  items. 

•  Software  Data  Dictionary  Document.  The  Software  Data  Dictionary 
Document  is  a  technical  document  prepared  for  the  programmers  and  <lata 
l>a.se  administrators.  It  provides  for  the  central  collection  of  information  about 


all  data  used  by  the  software  system:  all  files,  all  record  types,  all  items  within 
records,  all  relationships  between  records,  and  all  pertinent  information  about 
the  use  of  the  data.  The  Software  Data  Dictionary  Document  is  designed  to 
provide  a  standard,  consistent,  simple  framework  for  information  about  i  he  data 
userl  by  the  system  being  developed. 

•  Software  Design  Document  (SDD).  The  .Software  Design  Document  (SDD) 
describes  the  complete  design  of  the  each  (.'SCI.  It  describes  the  (  SCI  as  com¬ 
posed  of  Computer  Software  Components  (CSCs)  and  Computer  Software  Cnits 
(CSCs). 

•  Software  Development  Plan  (SDP).  The  Software  Development  Plan 
(SDP)  describes  a  contractor's  plans  for  conducting  software  development.  The 
SDP  is  used  to  provide  the  Government  insight  into  the  organizaticn(s)  respon- 
''ilde  for  pertorming  software  development  and  the  methods  and  procedures  ro 
be  followed  by  these  organization(s).  The  SDP  is  used  by  the  Government  ti; 
monitor  the  procedures,  management,  and  contract  work  effort  of  he  organiza¬ 
tion  performing  software  development. 

•  software  Product  Specification  (SPS).  The  Software  Product  Specifica¬ 
tion  (SPS)  consists  of  the  SDD  and  source  code  listings  for  a  ('SCI. 

•  Software  Programmer’s  ^^anuai  (SPM).  The  Softwai  Programmer  s 
Manual  (SP.M)  provides  information  needed  by  a  programmer  to  understand  the 
instiiicfion  set  a  chitecture  of  the  specific  host  or  target  computers.  The  SPM 
provides  information  that  may  be  used  to  interpret,  check  out.  troubleshoot,  or 
modify  existing  software  on  the  host  or  target  computers. 

•  Software  Quali.y  Program  Plan  (SQPP).  The  Software  Quality  Program 
Plan  (S(^PP)  identifies  the  organizations  and  procedures  to  be  used  by  the  con- 
tr^.ctor  to  perform  activities  related  to  the  Software  Quality  Program  specified 
bv  D0D-STD-2I6S.  The  SQPP  is  used  to  evaluate  the  contractor's  plans  fm 
implementing  the  Software  Quality  Program. 

•  Software  Test  Plan  (STP).  The  Software  Test  Plan  (STP)  describes  the 
lormal  qualification  test  plans  for  acceptance  testing  of  the  system.  The  S'l'P 
irlentifies  the  software  te^it  environment  resources  rerpiired  for  accreditation  te-.r- 
ing.  The  STP  identifies  the  individual  tests  that  will  be  performed  duriuH  ac¬ 
creditation  resting. 

•  Software  Test  Description  (STD).  The  Software  Test  Descripuo  describes 
each  of  the  procedures  identified  in  the  the  STP. 

•  Software  Test  Report  (STR).  The  Software  Test  Report  (STR)  is  a  record 
of  the  formal  f-iualification  testing  performed  on  the  system.  The  STR  provides 
the  Ciovernment  with  a  permanent  record  of  FQT  performed  on  the  system. 


Software  Users  Manual  (SUM  or  UM).  The  Software  L’ser's  Manual 
(Sl’M)  provides  the  user  personnel  with  instructions  sufficient  to  run  the  system. 

System  Operational  Concept  Document  (SOC).  The  System  Opera¬ 
tional  Coiu  ept  Document  describes  the  missioti  of  t  he  system  and  its  operational 
and  support  environments.  Iso  described  are  the  functions  and  characteristics 
of  the  computer  svstem  within  the  overall  system. 

System  Requirements  Specification  (SRS).  The  Software  Reriuirements 
Specification  (SRS)  specifies  the  engineering  and  ((ualification  reciuiremenls  for 
the  system. 

System/Segment  Design  Document  (SSDD).  The  System/Segment  De¬ 
sign  Document  (SSDD)  describes  the  design  of  the  system  and  its  operational 
and  support  environments.  It  describes  the  organization  of  the  system  as  com- 
po.sed  of  Hardware  Configuration  Items  (HW('Is).  Computer  Software  (  oiifig- 
u ration  Items  iCSCIs).  atid  manual  operations. 

System/Segment  Specification  (SSS).  The  System/Segment  Specification 
(  SSS)  specifies  the  recpiirements  for  a  system  or  a  segment  of  a  system.  The  SSS. 
upon  formal  approval.  Ijecomes  part  of  the  Functional  (or  lest/ Development  i 
Ba.seline. 

Version  Description  Document  (VDD).  The  Version  Description  Docu¬ 
ment  ( VDD)  identifies  and  describes  a  version  of  a  Computer  Software  Confiti- 
uration  Item  ((.'SCI)  being  released. 


C.  A  Guide  to  CASE  Tools 


Althoiigl)  tlii.s  list  will  (|\ucklv  bocotiie  oiilclated.  the  impact  of  Computer- Aidt'cl  Soft  ¬ 
ware  I'liigineeriiig  (CASh)  of  WVV  activities  is  too  important  to  ignore.  This  manual 
has  argued  for  the  importance  of  formality  in  .system  specification  and  design.  .\n 
atldil  ional  heiiefit  of  formality  is  that  formal  representations  can  be  manipulated  bv 
computers,  resulting  in  better  (luality  than  would  likely  be  obtained  manually.  .\t  a 
minimum,  certain  obvious  "clerirar’  errors  can  be  found  automatically,  such  as  incon¬ 
sistencies  in  interface  definitions.  Present  generation  (.’.-\.SE  tools  are  far  frcmi  idcsd. 
but  you  may  find  something  in  the  list  below — based  on  a  list  compiled  by  the  ('.\.Sb 
Research  Croup  of  Florida  .\tlantic  Cniversity — that  will  prove  very  u.sefnl  on  your 
l)rojecf . 

Adpac  Corp.  .\dpac  C.A.SF  Tools;  Brannan  St..  San  Francisco.  (  .'  .  'IDT.  1 1  T 
D7  l-(i6DD 

Advanced  Logical  Software  .\uatool;  D903  Santa  Monica  Blvd..  suite  108.  Beverly 
Hills.  CA  00212.  2l3-6r):5-.')786 

Advanced  Technology  International,  Inc.  SuperCase 

AGS  Management  Systems,  Inc.  Multi/CAM;  category:  front  end:  880  First 
.Ave..  King  of  Prussia.  P.\  19406.  21.5-265-1550 

.American  Management  Systems,  Inc.  Life  Cycle  Productivity  System:  categorv: 
front  end.  back  end:  1777  .Xorth  Kent  St..  Arlington.  V.A  22209.  703-841-0000 

Applied  Business  Technology  Corp.  Project  Workbench:  361  Broadwav.  .\ew 
York.  .NW  10013.  212-219-8945 

-Applied  Data  Research,  Inc.  DEPICTOR;  category:  front  end:  Route  200  and 
Orchard  Rd..  CX-8.  Princeton.  .N’.J  08543 

-Arthur  Andersen  &  Co.  Design/1  (part  of  Foundation  Series);  category:  front 
end.  back  end,  RE/.M;  33  West  .Monroe  St.,  Chicago.  IL  60603;  69  West  Wash¬ 
ington.  Chicago.  IL  60602.  312-580-0069.  312-580-0033.  312-507-5161 

-Atherton  Technology  Software  Backplane;  1333  Bordeau.x  Drive,  Sunnyvale.  (  .\. 
94089.  Tele:  408  734-9822.  Fax;  408  744-1607 

ASYST  Technologies,  Inc.  The  Developer;  One  .Naperville  Plaza,  Naperville.  (I. 
60540.  800-361-3673 

Bachman  Information  Systems  BACHMAN  Product  Set 

Cadre  Technologies,  Inc  Teamwork  OS/2  3.0;  category:  front  end;  222  Richmond 
St..  Providence,  RI  02903.  401-351-5950.  -I0l-351-C.\SE 


The  CADWARE  Group,  Ltd  SMA’A  Series;  category;  Front  end 
CASET  IPSYS  tool  Building  Kit:  7l4-496-8<>70 

Case  Ware,  Inc  \MPLIF\’:  4530  Hyland  Avenue,  Suite  1 15,  Costa  Mesa.  CA  '.)2(vJd. 
714-754-0408 

The  Catalyst  Group  PATHVC  Series;  category;  RE/M;  Peat  Marwick  Main.V 
Co.,  404  East  Wacker  Dr.,  Chicago,  IL  60601,  800-424-4059.  412-948-5.452 

CGI  Systems,  Inc.  P.\CBase,  P.ACBeuch.  P.\CDesign:  category:  front  (mkI,  hack 
end,  RE/M;  8200  Greensboro  Dr.  Suite  iOlO.  .McLean.  V.\  22102.  704-448-8181: 

1  Blue  Hill  Plaza,  Pearl  River.  .\’V  10965.  914-745-5040 

Chen<S<:  Associates  ER-Designer  (ERD);  1884  Constitution  Ave.  Ste  IE.  Babui 
Rouge.  L.\  70808.  504-928-5765 

Cinconi  Systems,  Inc.  Supra.  .Mantis.  Easy  PC  Contact.  C.\SE  Interchange:  2490 
.Montana  .Ave..  Cincinnati,  OH  45211.  800-888-0115 

Coding  Factory  CoFac 

Cognos  Powercase;  67  S.  Bedford  St..  Burlington.  Mass.  01804.  617-229-6600 

Computer  Associates  International,  Inc.  C.A-Dataconi.  C.A-Ideal.  C.A-Dataquerv. 
C.A-Dataquery  PC;  Computer  .Associates  World  Headquarters.  711  Stewart 
•Ave.,  Garden  City.  N'Y  11540.  516-227-4400 

Computer  Data  Systems  Scan/COBOL.  Superstructure;  1  CurieCourt.  Ro<  kville. 
MD  20850.  202-921-7000 


Computer  Sciences  Corp  Design  Generator:  category;  front  end:  4610  Fairview 
Park  Dr.  Falls  Church.  V.A  22042.  704-876-1000 

Computer  Systems  Advisers,  Inc  POSE  4.0;  50  Tice  BlvcL.  WooclclifF  Lake.  .\.) 
07675.  800-5.37-4262,  201-491-6.500 

Compuware  Corporation  C.ATI  tools:  .Abend-.AID.  CK/S  .Abend-.AID.  CICS  R.\D.\R. 
File-.AID  family,  TransREL.ATE.  PL.AYB.ACK.  File  PL.AYB.ACK.  SIMCLC.VST. 
dBLG-.AID,  .XPEDITER.  N.A\  IG.A TOR;  41 140  .Northwestern  Highway,  Fnriii- 
ington  Hills.  Michigan  48018-5.550 

Cortex  Corp,  CorVision,  .Application  Factory;  category:  front  end.  back  end.  RE,  .M; 

148  Technology  Dr.,  Waltham.  M.-\  021-54;  100  Fifth  .Avenue,  Waltham.  M.A 
021-54-9864,  617-894-7000 

Cullinet  Software,  Inc.  ID.MS/.Architect 

D.  Appleton  Company  I DEF/ Leverage:  1444  Park  V  iew  .Ave..  Suite  220.  Man¬ 
hattan  Beach.  C.A  90266.  214-5-16- 7575 


Deft  Inc.  Deft;  567  Dixon  Rcl..  suite  1  IQ,  Rexclale.  ON  M9VV  IHT,  Canada,  416- 
249-2246 

Deloitte,  Haskins^^  Sells  4Front;  200  East  Randolph  Dr..  Chicago.  IL  60601.  412- 
5!o6-816S 

Digital  Equipment  Corp.  DECASE;  DECdirect.  Continental  Blvd..  .Merrimack. 
NH  0:30.54.  S00-;344-4S2.5 

ECS  Associates  SQL-Link-Plus;  ;3S12  Sepulveda  Blvd..  Torrance.  C.A  90.50.5.  213- 
3TS-9260 


ICONIX  Software  Engineering  Inc.  PovverTools  Series;  category:  front  end,  hack 
end,  RE/M:  2S00  Twenty  Eighth  St.  Suite  320.  Santa  Clara,  C.\  9040.5.  213- 
4.5S-0092 

Forschungszentrum  Informatik  (FZI)  STONE:  Haid-und-.Neu-Str.  10-14..  D- 
7500  Karlsruhe.  Germany.  4-49-721-6906-731 

i-Logix  State.Mate;  22  Third  .Ave..  Burlington.  .M.A  01S03.  617-272-S090 

Index  Technology  Corp.  Excelerator  1.84:  category;  front  end:  One  .Main  St.. 
Cambridge,  .MA  02142,  SOO- 777-8858.  617-494-8200 

Institute  for  Information  Industry  KangaTool  Series;  category:  front-end:  8th 
Floor.  106  Ho-Ping  E.  Rd..  Taipei.  Taiwan.  R.O.C. 

Integrated  Systems,  Inc.  .AutoCode:  2500  Mission  College  Blvd..  Santa  Clara.  (^.A 
9.50.54.  408-980-1500 

Interactive  Development  Environments  Software  Through  Pictures;  category: 
front  end;  595  .Vlarket  St.,  12th  Floor,  .San  Francisco,  C.A  94105.  115-543-0900 

Knowledge  Ware,  Inc.  lEW/VVS:  category;  front  end;  3340  Peachtree  Rd..  .At¬ 
lanta.  GA  :30026,  404-231-8575.  800-:3;3'8-4130 

Language  Technology  RECODER,  INSPECTOR:  category:  RE/M:  27  Congress 
St.  Salem.  MA  01970,  800-7.32-6;3.37.  .508-741-1507 

Learmonth^  Burchett  Management  Systems,  Inc.  (LBMS)  System  Engineer 
(nee  .Auto-Mate  Plus);  1800  West  Loop  South.  Suite  1800,  Houston,  TX  77027. 

7 1 3-682-8530.  800- 23 1-751 5 

Manager  Software  Products,  Inc.  Manager  Series;  category:  Front  end.  back 
end;  131  Hartwell  Ave,  Lexington.  .M.A  02173-3126.  617-86:3-5800 


Matterhorn,  Inc,  HIBOL;  category;  back  end 


McDonnell-Douglas  ProKif' Workbench  STRADIS.  PRO- IV';  category:  front  end: 

P.O.  Box  ol6.  Dept.  Lolo.  MS  2812301.  St.  I.oui.s.  MO  63106.  800-325- 1087. 
800-822-7337.  3 13-232-57 1 5 

Mentor  Graphics  Corp.  .Analyst/RT.  Designer.  .Auditor:  category:  front  (mkI:  8500 
Southwest  Creekside  Place,  Beaverton,  OR  07005.  503-626-7000 

Meta  Systems  QuickSpec.  Structured  .Vrchitect  (S.A ).  Struct  ured  .\rchit('(  t-Integraloi 
(S.A-I).  PSL/PS.A.  Report  Specification  Interface  (RSI).  View  Integration  Sys¬ 
tem  (\  IS);  category:  front  end.  RE/.M:  315  E.  Eisenhower  Parkway.  Suite  200. 

.\nn  .Arbor.  MI  48108,  313-663-6027 

Micro  Focus,  Inc.  (^OBOL/2  Workbench:  2365  East  Bayshore  Rd..  Palo  Alto.  O A 
03.303.  U.5-8.56-3L61 

Netron,  Inc.  .\’ETROi\'/C'.AP;  00  St.  Regis  Crescent  .\'.  Downsview.  Ontario,  ('anada 
M3.I  n’O.  416-636-8333 

On-Line  Software  International  ('a.sePac;  2  Executive  Dr..  Et.  Lee  Executive 
Park.  Ft.  Lee.  .NM  07024.  201-502-0009 

Optima,  Inc.  Design  Vision  1.7.  Design.Machine  2.0:  category;  front  end.  back  etid 

Oracle  Systems  Corp.  CASE'Designer.  C.ASE*Dictionary.  C.ASE ‘Generator.  SQL* Forms. 
SQL'Report.  SQL*QMX.  Oracle.  SQL'Louder;  Oracle  World  Headf|uarters. 

500  Oracle  Pkwy.  Redwood  Shores.  C.A  94065.  315-506-7000;  OR.ACLE  Corpo¬ 
ration.  20  Davis  Drive.  Belmont.  C.A  94002.  800-345-DBMS 

Pansophic  Systems  Inc.  Telon;  2400  Cabot  Drive.  Lisle.  IL  ()0532.  312-505-(>000. 
>00-323-7335 

Phoenix  Technologies,  Ltd.  P-.Source.  P-Tools;  846  I'niversitv  .\ve..  .Norwood. 

.M.A  02062.  617-551-4000 

Popkin  Software<Si  Systems  System  .Architect;  111  Prospect  St..  Suite  505.  Stam¬ 
ford.  CT  06901.  203-323-3434 

ProMod,  Inc.  ProMod  Series;  category;  front  end.  backend.  RE/.M:  23685  Birtcher 
Dr..  El  Toro.  C.A  92630.  713-8.5.5-3036.  800-255-2689 

Rational  Rational  Design  Facility;  category:  front  end;  3320  Scott  Blvd.  Santa 
Clara.  C.A  95054 


Ready  Systems  Corp.  CardTools:  470  Potrero  .Ave..  P.O.  Box  60217.  Sunnyvale. 
C.A  94086 

Sage  Software  Inc.  Polytron  Version  Control  System  (PVCS).  .APS  Development 
Center:  category;  back  end.  RE/M;  1700  .N.W.  167th  Place.  Beaverton.  OR 
97006.  800-547- 1000 


Sapiens  International  Perfect.  Object-Modeller.  Sapiens.  Quix:  Sapiens  I  SA.  J!)o 
7th  Ave..  New  York.  NY  lOOOl.  212-366-9394 

Schemacode  International  Inc  Schemacode.  Datrix;  89  OU'enbrooke.  snile  lUO. 
Dollard  des  Ormeaux.  Quebec  H9A  2L7.  51  1-683-8693.  fax  51  1-683-6792.  da- 
t  rix  Qrgl.polymtl.ca 

Six  Sigma  Case  Canonizer;  13456  SE  27th  Place.  Belleviu*.  VVA  98005.  296-613- 
6911 

Softlab,  Inc.  .Maestro;  category:  front  end,  backend.  RE/.M;  188  The  Einbai  (  adeio. 
Bayside  Plaza.  Suite  750,  San  Francisco.  CA  94105.  115-957-9175 

Software  AG  of  North  America,  Inc.  Adabas,  Natural.  ( 'onstruct.  Preflici.  Pre¬ 
dict  Case.  Super  Natural:  11190  Sunrise  Valley  Drive,  Restoii.  V’A  22091.  793- 
860-5050 


Software  Architecture  and  Engineering  Strategic  .Networked  .Xpplication  Plat¬ 
form;  1600  Wilson  Blvd..  .Arlington,  V.A  22209,  703-276-7910 

StarSys,  Inc.  .MacBubbles:  category:  front  end:  11113  Norlec  Dr..  Silver  Spriiur. 
MD  20902 

Syscorp  International,  Inc.  MicroStep  1.3:  9420  Research  Blvd..  Suite  200.  .\ubtin. 
T.X  787.59.  512-338-0.591 

Telelogic  Europe  SDT:  33  Boulevard  de  la  Cambre.  B-1050  Brus.sels.  Belgium.  011- 
32-2-647-3670 

Texas  Instruments  Inc.  Information  Engineering  Facility  (lEF)  4.0:  6550  ('base 
Oaks  Blvd.,  Plano.  T.X  75023.  800-527-3500 

Tom  Software  .Application  Xcellence;  127  SW  l.5^‘h  Street,  Seattle.  \V.A  '.'8166. 
206-246-7022 

Tranform  Logic  Inc.  (Previosly  Nastec  Corp.)  Design.Aid  4.3:  category:  front 
end:  24681  Northwestern  Hwy.,  Southfield.  MI  48075.  800-872-8296  7799  Lees¬ 
burg,  Suite  1110.  North  Tower.  Falls  Church.  V.A  22043,  703-556-9401 

Transform  Logic  Corporation  Transform;  8502  East  Via  de  Aentura.  Scottsrlale. 
AZ  852.58.  602-948-2600 

Unisys  Corp.  Line  Design  .Assistant.  Line,  .Mapper,  DMS  II:  P.O.  Box  500.  Bluebell. 
PA  19424,  21.5-986-4011 

ViaSoft,  Inc.  Via/Insight.  Via/SmarTest;  3033  North  44th  St..  Suite  280.  Phoenix. 
AZ  8.5018.  602-9.52-00.50 

Visible  Systems  Corp.  Visible  Analyst  Workbench;  category;  front  end:  9-50  Win¬ 
ter  St..  Waltham.  M.A  02154.  617-969-4100 


Visual  Software,  Inc.  vsDesigner.  vsSQL.  vsObject  Maker:  category:  Iroiit  eiul; 
Freedom  Ciirle.  Suite  oAO.  Santa  Clara.  CA  95054.  408-988-7575 

Westmount  Technology  B.V.  ISEK.  TSEE,  RTEE;  5020  l  lSrh  Ave  N  F...  P.O 
Box  97002.  Bedmond.  \\  A  98074-9702 

Yourdan,  Inc.  A  iialvst/ Designer  Toolkit.  Cratlle;  category:  front  end:  1501  Broad- 
wav,  New  5ork.  N5  10040,  212-491-2828 


ROME  LABORATORY 


Rome  Laboratory  pleats  and  executes  an  interdisciplinary  program  in  re- 
search^  development,  test,  and  technology  transition  in  support  of  Air 
Force  Command,  Control,  Communications  and  Intelligence  (C  D  activities 
for  all  Air  Force  platforms.  It  also  executes  selected  acquisition  programs 
in  several  areas  of  expertise.  Technical  and  engineering  support  within 
areas  of  competence  is  provided  to  BSD  Program  Offices  (POs)  and  other 
BSD  elements  to  perform  effective  acquisition  of  C  l  systems,  bi  addition, 
Rome  Laboratory's  technology  supports  other  AF9C  Product  Divisions,  the 
Air  Force  user  community,  and  other  DOD  and  nonrDOD  agencies.  Rome 
Laboratory  maintains  technical  competence  and  research  programs  in  areas 
incluidtng,  but  not  limited  to,  communications,  command  and  control,  battle 
management,  intelligence  information  processing,  computational  sciences 
and  soft-ware  prOdicibility,  wide  area  surveillance/sensors,  signal  proces¬ 
sing,  solid  state  sciences,  photonics,  electromagnetic  technology,  super¬ 
conductivity,  and  electronic  reliability/maintainability  and  testability. 


